{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "dcea48ed",
   "metadata": {},
   "source": [
    "over fitting，过拟合，只对某个数据集过度拟合的状态\n",
    "神经网络以某个指标为线索寻找最优权重参数，神经网络的学习中所用的指标称为损失函数(loss function)\n",
    "损失函数是表示神经网络性能的\"恶劣程度\",即当前神经网络对监督数据在多大程度上不拟合、不一致\n",
    "损失函数可以是任意函数，一般使用均方误差和交叉熵差等\n",
    "\n",
    "### 均方误差(mean squared error):\n",
    "$$\n",
    "E = \\frac{1}{2}\\sum_{k}^{}(y_{k}-t_{k})^{2}   \\quad \\text{(4.1)}\n",
    "$$\n",
    "\n",
    "$y_{k}$是表示神经网络的输出,$t_{k}$表示监督数据，k表示数据的维数\n",
    "\n",
    "### 交叉熵误差(cross entropy error)\n",
    "$$\n",
    "E = -\\sum_{k}t_{k}logy_{k}   \\quad \\text{(4.2)}\n",
    "$$\n",
    "log表示以e为底数的自然对数($log_{e}$),$y_{k}$是神经 网络的输出，$t_{k}$是正确解标签，并且$t_{k}$中只有正确解标签的索引为1，其他均为0  \n",
    "(正确解标签表示为1，其他均为0的表示方法称为**one-hot表示**)  \n",
    "式(4.2)实际上只计算对应正确解标签的输出的自然对数，假设正确解标签的索引是2，则交叉熵误差=-log 0.6 = 0.51  \n",
    "\n",
    "### 批次训练\n",
    "$$\n",
    "E = - \\frac{1}{N}\\sum_{n}\\sum_{k}t_{nk}logy_{nk} \\quad \\text{(4.3)}\n",
    "$$\n",
    "假设数据有N个，$t_{nk}$表示第n个数据的第k个元素的值($y_{nk}$是神经网络的输出，$t_{nk}$是监督数据)  \n",
    "### 导数  \n",
    "$$\n",
    " \\frac{df(x)}{dx} = \\lim_{h\\to0}\\frac{f(x+h) - f(x)}{g} \\quad \\text{(4.4)}\n",
    "$$\n",
    "导数的含义是'x'的“微小变换”将导致函数f(x)的值在多大程度上发生变化  \n",
    "### 偏导数  \n",
    "\n",
    "### 梯度  \n",
    "梯度指示的方向是各点处的函数值减小最多的方向，但不一定指向最小值  \n",
    "梯度法：通过不断的沿梯度方向前进，逐渐减小函数值的过程  \n",
    "$$\n",
    "x_{0} = x_{0} - η\\frac{∂f}{∂x_{0}}  \\quad\n",
    "x_{1} = x_{1} - η\\frac{∂f}{∂x_{1}}\n",
    "$$\n",
    "η表示更新量，在神经网络学习中，称为学习率(learing rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "358e3a18",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiIAAAGdCAYAAAAvwBgXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAy/klEQVR4nO3deXxV9Z3/8ffNdrPfLBeykJuEsCM7KEtV1FqX2lHHamXquLRKy2+GKcqjdWC0io4dpoU6MzpqN4rOVC1tXcelQl0oisgWFNkhQBJCCEnIvVlvlnt+fyS5EJaQG3Luyc19PR+P80hyc3LPJ99H5Lz9bsdmGIYhAAAAC0RYXQAAAAhfBBEAAGAZgggAALAMQQQAAFiGIAIAACxDEAEAAJYhiAAAAMsQRAAAgGWirC6gOz6fT2VlZUpKSpLNZrO6HAAA0AOGYai2tlbZ2dmKiOi+z6NfB5GysjK5XC6rywAAAL1QUlKinJycbs/p10EkKSlJUvsvkpycbHE1AACgJzwej1wul/8+3p1+HUQ6h2OSk5MJIgAAhJieTKtgsioAALAMQQQAAFiGIAIAACxDEAEAAJYhiAAAAMsQRAAAgGUIIgAAwDIEEQAAYBmCCAAAsExQgsizzz6roUOHKjY2VlOnTtW6deuCcVkAANDPmR5EVq1apfvvv18PPfSQCgsLddlll+n6669XcXGx2ZcGAAD9nM0wDMPMC0yfPl1TpkzRc889539tzJgxuvnmm7V06dJuf9bj8cjhcMjtdvOsGQAAQkQg929TH3rX3NysLVu2aNGiRV1ev+aaa7R+/fozzvd6vfJ6vf6vPR6PmeUBABA2Wtp8qqj1qtzdqKPuJpW7m3TU3aR8Z4LunJFnWV2mBpHKykq1tbUpIyOjy+sZGRkqLy8/4/ylS5fqscceM7MkAAAGnOZWn4552oPFUXejymqaTgaOjtcr67w62xjIZSOcAzeIdDr9McCGYZz10cCLFy/WwoUL/V97PB65XC7T6wMAoL9qaTsZMspqTvZmlNU0qtzTpLKa9pDRE9GRNmUkxyrLEatMR5yyHLEak5Vk8m/QPVODiNPpVGRk5Bm9HxUVFWf0kkiS3W6X3W43syQAAPoNwzBUVd+sspr2Xoz2j+1ho8zdqKM1TaqobZKvB7M5Y6IilOWI7TjilOmIVfYpgSPTEau0+BhFRJzZEWAlU4NITEyMpk6dqjVr1uhv//Zv/a+vWbNGN910k5mXBgDAck0tbSqradSRjoBx5JSwUVbTqDJ3k5pbfed9n+hImzI7Aka2I1ZZKXFdQkaWI1ZpCTFnHW3o70wfmlm4cKHuvPNOTZs2TTNnztSvfvUrFRcXa968eWZfGgAA0xiGoZqGFh2paVTpiVPCxolGlbnbP1bVN5/3fWw2aVCiXdkpcRqS0hEsOoJGdkqcslJi5Uyw97uejL5iehC5/fbbVVVVpccff1xHjx7VuHHj9M477ygvz7qJMQAAnI/PZ+h4nVelJxpVeqJBRzpCxqkfG5rbzvs+CTGR7SEjNU5ZjjgNSWkPGNkpccruGEKJiQrfjc5N30fkQrCPCADALD6foYpar0pPNPjDRvvHk2Gjue38wyaDkuwa0tGbMSS1/WN70IhVTkq8kuOiQnLI5EL0m31EAACwimEYqqxrVklHwCip7ho4ehI0IiNsykyO1ZDUOOV0BI2c1DgNSYnv6OGIVWx0ZJB+o4GJIAIACFm1TS0qqW5UyYkGlVR3HKeEjsaW7odOOoOGKy1OOanxykk99WOcMpNjFRUZvsMmwUAQAQD0W20+Q0fdjSqualBx9cmjM3BUn2cyqM2m9qCRGq+cjrDh6ggbrjSCRn9AEAEAWKre23oyZFQ16HB1vYqrG1VcVa8jNY1qaet+KmNqfLRy0+KVkxYvV0fAyO34PDslLqwngoYCgggAwFSdy1wPVdXrcFVD+1F98vPz7QoaHWlTTmq8ctNOHu1hI0GutDglxUYH6TeBGQgiAIAL1jkx9HBVvQ5VNehwVb0OVnaGjXp5mlq7/XlHXLTy0ttDRl56vPLSEuTq+DwjOVaRA3QPDRBEAAA9ZBiGquubdbCyPWQc6ggdhzoCR523+7CRmRyr3PR45afHKy89oUvocMTTqxGuCCIAgC48TS061BE2io63B47O8FHbTc+GzSZlO+KU74xXfnqC8tMTlJser6HO9tDBMlecDUEEAMJQc6tPxdUNKjpe5w8cByvrVVRZp8q6c69EOT1sDHUmKC89QUOd8XKlxcseRdhAYAgiADBAdT7Z9UBFnYoq6/0fi47XqeREo9q6eaTroCS7hjoTNDQ9QUMHtfduFAyiZwN9jyACACGupa29d2N/RZ2KjtfrwPG69qOirttJovExkSoYlKChzkQVONuDRoEzUfnOeFaiIGgIIgAQIhqaW3Wgol77j9dqf0Wd9lfU6cDxeh2uqj/nXhs2mzQkJU7DBiW2B41BiRrmbP+YkWwPu2egoP8hiABAP+NuaNH+47Xad6xO+zoCx/6KOh2paTznz8RFR2rY4AQNG5R48hjcPqTCUAr6M4IIAFjE3dCifRW12nusTnuPtfdy7D1Wq4rac2/wlZ4Qo2GDEzV8cHvYGN7xeVZyrCLYawMhiCACACarbWrRvoo67S0/GTrOFziyHLH+kDFicJL/87SEmCBWDpiPIAIAfcTb2qYDFfXae6xWu8trtafco73Huh9SyXbEanhGkkYOTtTIjCQNz0jUiMGJTBZF2CCIAECAfD5DR2oateuoR3vKa7X7WK32lNfqYGX9OZfEDk6ya1RmkkYMTtKozESNyEgicAAiiABAt+q8rdpT7tHOo7XafdTT0dNRe87tzJNjozQ6M1kjMxM1KiNJIzOSNCozSSnxDKkAZ0MQAQC1b/5VeqJRO496tMt/1Kq4uuGs50dH2jRsUKLGZCVrVGZ72BidmaTM5FiWxAIBIIgACDvNrT7tq6jVjjKPdpZ5/OHjXM9RyUi2a3RmssZkJWtMVpJGZyarYFCCoiMjglw5MPAQRAAMaHXeVu066tGOI27tKPNoR5lH+ypqz7oBWHSkTSMGJ2lsdrJGZyZpbFayRmcls1IFMBFBBMCA4W5o0Y4yt74sc2v7kfbwcbCqXsZZ5o8mx0ZpbHayxmY5NDY7WRdlJ2vYoETFRNHLAQQTQQRASHI3tGj7Ebe2H3Hry46P55rPkZkcq4s6wsbYbIcuyk5WTmocczmAfoAgAqDfq/O2anupW9uP1OiL0vbQcbjq7KHDlRan8UMcuijboXFD2kOHM9Ee5IoB9BRBBEC/4m1t066jtfq8pEafl7YHjwPH6846vJKbFq/xOQ6NH9J+jMt2yBHPvhxAKCGIALCMz2eoqLLeHzo+L6nRzqOes04kHZISpwk5Do3PcWjCkBSNH0LoAAYCggiAoDlR36xtJTUqLD6hwpL24OE5y5LZtIQYTchxaGJOiia6HJqQk8LwCjBAEUQAmKK1zac9x2q1tbg9eGwrrlFRZf0Z59mjIjR+iEOTXCma6ErRJFcKE0mBMEIQAdAn3A0t2lpyQlsPn9CWwyf0eUmN6pvbzjivYFCCJrlSNDk3VZNdKRqVmcTGYEAYI4gACJhhGCqubtCmQ+2hY8vhau09VnfGeUn2KE3K7QgduSma7ErhmSsAuiCIADiv1jafdpfXauPBam0+XK1Nh07oeK33jPOGOhM0JTdVU/Paj+GDExUZwRALgHMjiAA4Q1NLmz4vqdGmQ9XaeKh9uOX0p83GREZofI5D0zpCx5S8VCaUAggYQQSAGpvbtLX4hD4rqtKGg9XaVlKj5lZfl3OS7FGalp+qaflpujg/TRNyHIqNjrSoYgADBUEECEONzW3acviENhRVaUNRlT4vrTlj7w5nol3Th6bp4vxUXTw0TaMzkxlmAdDnCCJAGPC2tqmwuEbrD1Tp0wOV2lZyZvDIdsRqekG6LhmapulD0zTUmcASWgCmI4gAA1Cbz9CXR9z6eH+lPj1QpU2HquU9baglyxGrmQXpmtFxuNLYuwNA8BFEgAHAMAwdrKzXx/sr9fG+Sm0oqjpjx1Jnol2zhqVr5rB0zSxIV156PMEDgOUIIkCIOlHfrE8OtAePdfsqdaSmscv3k2KjNLMgXV8Z7tSsYekaPjiR4AGg3zE1iPzkJz/R22+/rW3btikmJkY1NTVmXg4Y0FrbfNpWUqO/7j2utfsq9UVpTZcn0kZH2jQ1L1WXDnfq0hGDNC47WVHsWAqgnzM1iDQ3N+u2227TzJkztWLFCjMvBQxI5e4mrd1boY/2HNfH+ytVe9pwy8iMRF02YpAuHeHU9KFpio+hkxNAaDH1X63HHntMkvT888+beRlgwGht82lrcY0+3FOhD3dXaHd5bZfvp8RH69LhTl0+cpAuHzFImY5YiyoFgL7Rr/73yev1yus9uW20x+OxsBogOE7UN2vt3uN6f3eF/rr3uNyNLf7v2WzShJwUXTFykK4YNUgTclLYywPAgNKvgsjSpUv9vSjAQGUYhg4cr9f7u47p/V0V2ny4Wr5T5nqkxEdr9shBunLUYF02wql0tk0HMIAFHESWLFly3rCwadMmTZs2LeBiFi9erIULF/q/9ng8crlcAb8P0N+0+QxtLT6hNTuPac3OYzpYWd/l+6Mzk3TV6MG6avRgTXKlMMkUQNgIOIjMnz9fc+bM6fac/Pz8XhVjt9tlt/N/fxgYmlra9PG+Sq3eWa73d1Woqr7Z/72YyAhNL0jT1WMy9NUxg5WTGm9hpQBgnYCDiNPplNPpNKMWIOTVNrXog90VWr3jmD7cU6GG5jb/9xxx0bpq9GBdPSZDs0cNUqK9X42MAoAlTP2XsLi4WNXV1SouLlZbW5u2bdsmSRo+fLgSExPNvDQQNO7GFv1l5zG9s/2o1u2rVHPbya3UsxyxuvaiTF1zUYYuzk9TNEMuANCFqUHkkUce0QsvvOD/evLkyZKkDz/8UFdccYWZlwZM5W5o0eqd5Xp7+1F9sr+yywPkCpwJum5cpq4bl6nxQxzsZgoA3bAZhmGc/zRreDweORwOud1uJScnW10OwlxtU4vW7Dymt784qr/uO94lfIzKSNL14zP19fFZGsFW6gDCXCD3bwapgW40tbTpw90VemNbmT7YU6HmU55gOyojSTdMyNLXx2dp+GCGGgGgNwgiwGnafIbWH6jU64Vlem9Hueq8J7dVHzYoQd+YkK1vTMjSiIwkC6sEgIGBIAKofZOxXUdr9Vphqd7YVqaK2pM7/A5JidM3JmbppolDNCYriWEXAOhDBBGEtYraJr1RWKZXtpZ2ea5LSny0bhifpZsnD9HU3FRFsK06AJiCIIKw421t0/u7KvTKllJ9tPe42jr2V4+JitDVYwbr5klDdMWowYqJYqktAJiNIIKwsbvco1WbSvR64RGdaDj5YLkpuSn65tQcfWN8thzx0RZWCADhhyCCAa3O26o3t5Vp1eYSfV5S4389MzlWt0wZom9OzdGwQax4AQCrEEQwIH15xK0XPyvWm9uOqL5jm/WoCJuuHpOh2y926fKRgxTJvA8AsBxBBANGY3Ob/u/zMv3us8P6otTtf73AmaA5l7h0y5QcORN5qCIA9CcEEYS8Q5X1+t2Gw/rjllK5G9vnfsRERui6cZn6u0tyNaMgjSW3ANBPEUQQknw+Q2v3HdfznxzS2r3H/a+70uL099PzdOvUHKXT+wEA/R5BBCGl3tuqV7eWauX6Qyo6Xi9Jstmk2SMH6a6ZeZo9cjBzPwAghBBEEBLKahr1/PpDenljsWqb2rdcT7JH6VsXu3TXzDzlpSdYXCEAoDcIIujXdpS59Zt1B/V/n5eptWPjsaHOBN0zK1/fnJqjRDt/wgAQyvhXHP2OYRj6ZH+VfrH2gD7eX+l/fWZBuu67bKiuHDWYLdcBYIAgiKDf8PkMrd5Zrmc/OuBffhsZYdPXx2fpe5cVaHyOw+IKAQB9jSACy7W2+fTGtjI9t/aA9lfUSZJioyM05+Jc3XvpULnS4i2uEABgFoIILNPS5tNrW4/ovz/cr+LqBklSUmyU7p6Zr+98JZ/ltwAQBggiCLqWNp9e2VKqZz7ar5LqRklSekKM7rusQH8/I1dJsTx4DgDCBUEEQdPmM/R64RH95/t7/QHEmRij718+THfMyFV8DH+OABBu+JcfpvP5DP15R7meXLPXPwfEmWjXvNkFumN6nuJiIi2uEABgFYIITPXxvkr9+5936csjHkmSIy5a82YP092z8ugBAQAQRGCOnWUeLX13l9bta98HJCEmUvdeVqD7LhuqZOaAAAA6EETQp8pqGrV89R69VnhEhiFFR9p0x/Q8/dNVw1kFAwA4A0EEfaKxuU2/WHtAv/zrATW1+CRJ35iQpR9dO4rnwAAAzokgggtiGIbe/LxM//7ubh11N0mSLslP00M3jNFEV4q1xQEA+j2CCHptR5lbj7yxQ1sOn5AkDUmJ00M3jNH14zJls/EsGADA+RFEEDBPU4ueXL1X//PpIfkMKT4mUv9wxTDdd1mBYqNZigsA6DmCCHqscxjmibd36XitV1L7PJCHbxirTEesxdUBAEIRQQQ9criqXv/y2nZ9sr9KklTgTNDjN43TpSOcFlcGAAhlBBF0q7XNp99+clBPrtmrphaf7FER+qerhmvu5QWyRzEMAwC4MAQRnNPOMo8WvfqFvih1S5JmDUvX0lvGsxwXANBnCCI4Q0ubT898uF///cF+tfoMJcVG6eEbxuhb01yshgEA9CmCCLrYX1GrhX/43N8Lct1FmXr8pos0OJnJqACAvkcQgaT2J+SuXH9IP/vzbnlbfXLERevxmy7SjROz6QUBAJiGIAId8zTp/t9v06dF7StiLh85SD/75gSW5AIATEcQCXMf7D6mH/7xC1XXNysuOlIPf2OMvn1JLr0gAICgiDDrjQ8dOqR7771XQ4cOVVxcnIYNG6ZHH31Uzc3NZl0SAfC2tulf39qp7z6/WdX1zRqblay3f3Cp7pieRwgBAASNaT0iu3fvls/n0y9/+UsNHz5cX375pebOnav6+notX77crMuiB4qrGvSPL23V9iPtE1LvmZWvxV8fzb4gAICgsxmGYQTrYsuWLdNzzz2noqKiHp3v8XjkcDjkdruVnJxscnXh4cPdFVrw+0J5mlqVEh+tZbdO1NfGZlhdFgBgAAnk/h3UOSJut1tpaWnn/L7X65XX6/V/7fF4glFWWPD5DD39wX795/t7ZRjS5NwUPfPtKcpOibO6NABAGDNtjsjpDhw4oKefflrz5s075zlLly6Vw+HwHy6XK1jlDWjuxhbN/Z/N+o+/tIeQv5+Rq99/bwYhBABguYCDyJIlS2Sz2bo9Nm/e3OVnysrKdN111+m2227Tfffdd873Xrx4sdxut/8oKSkJ/DdCFweO1+mm//5Y7++uUExUhJbdOkFP3Dye+SAAgH4h4DkilZWVqqys7Pac/Px8xca270FRVlamK6+8UtOnT9fzzz+viIieZx/miFyY9QcqNe9/t8jT1KohKXH65Z1TNW6Iw+qyAAADnKlzRJxOp5zOnj36/ciRI7ryyis1depUrVy5MqAQggvzh00l+pfXtqvVZ2hKbop+ddc0ORPtVpcFAEAXpk1WLSsr0xVXXKHc3FwtX75cx48f938vMzPTrMuGPZ/P0E/f261frm1fmXTjxGz97NYJio1mKAYA0P+YFkRWr16t/fv3a//+/crJyenyvSCuGA4rza0+LfzDNr31xVFJ0oKvjtD9V49ggzIAQL8V1H1EAsUckZ6r97Zq3u+2aN2+SkVH2rTs1om6efIQq8sCAIShfruPCMxxor5Z33l+k7aV1Cg+JlK/vHOqLhsxyOqyAAA4L4JIiCt3N+nOFZ9pX0WdUuKjtfKeizU5N9XqsgAA6BGCSAgrqW7QnF9t0JGaRmUk2/W/907XyIwkq8sCAKDHCCIh6tQQkp8er/+9d7pcafFWlwUAQEAIIiGo9ESD/u7X7SFkqDNBv//eDGUkx1pdFgAAAWOHsRBTVtOov/v1BpWeaO8JeXkuIQQAELoIIiHkqLtRc361QSXVjcpLj9fL35uhTAchBAAQuggiIeJEfbP+/jefqbi6Qa60OL08d4ayHDw9FwAQ2ggiIaCxuU33vrBJB47XK8sRq5fnzlB2CiEEABD6CCL9XGubT//0cqG2FtcoOTZKL3z3EuWksjoGADAwEET6McMw9OM3vtRfdh1TTFSEVtxzMfuEAAAGFIJIP/aff9mnlzeWKMImPTVnsi7OT7O6JAAA+hRBpJ96rbBU//X+PknSYzeN03XjMi2uCACAvkcQ6Ye2l7q16JXtkqT/d8Uw3Tkjz+KKAAAwB0Gkn6ms8+r7/7tZ3lafrho9WD+8ZpTVJQEAYBqCSD/S0ubTP764VWXuJhU4E/Qft09SZITN6rIAADANQaQf+cnbu/TZwWol2qP0q7umyhEXbXVJAACYiiDST/xpS6meX39IkvQft0/S8MEs0wUADHwEkX5gf0WdHn69fXLq/VeP0NfGZlhcEQAAwUEQsVhzq0/3rypUU4tPl41w6gdXjbC6JAAAgoYgYrEn1+zVl0c8So2P1vLbJiqCyakAgDBCELHQpweq9Mu/HpAkLb1lgjKSYy2uCACA4CKIWMTd0KKFf9gmw5DmXOxi51QAQFgiiFjAMAz9y2vbddTdpKHOBP34G2OtLgkAAEsQRCzw5udlenv7UUVF2PSft09Sgj3K6pIAALAEQSTI3I0t+te3dkqS/umqEZroSrG2IAAALEQQCbLl7+1RZV2zCgYlaN4VBVaXAwCApQgiQbStpEa/++ywJOmJm8fJHhVpcUUAAFiLIBIkrW0+PfTadhmGdMvkIZo1zGl1SQAAWI4gEiT/8+lh7SjzKDk2Sv9ywxirywEAoF8giARBubtJP1+9R5K06PoxcibaLa4IAID+gSASBP/61k7VN7dpcm6K5lzssrocAAD6DYKIyQqLT+jt7UcVYWufoMqzZAAAOIkgYrLlHUMyt0zJ0UXZDourAQCgfyGImGj9/kp9sr9K0ZE2LfjqCKvLAQCg3yGImMQwDC3r6A359iW5cqXFW1wRAAD9D0HEJO/vqlBhcY1ioyP0j1cNt7ocAAD6JYKICXw+wz835J5ZQzU4KdbiigAA6J9MDSI33nijcnNzFRsbq6ysLN15550qKysz85L9wlvbj2p3ea2S7FGaN5vnyQAAcC6mBpErr7xSf/jDH7Rnzx698sorOnDggG699VYzL2m51jaf/mPNXknS3MsLlBIfY3FFAAD0X1FmvvkDDzzg/zwvL0+LFi3SzTffrJaWFkVHR5t5acu8WnhEByvrlZYQo+9eOtTqcgAA6NdMDSKnqq6u1osvvqhZs2adM4R4vV55vV7/1x6PJ1jl9QnDMPTbjw9Kkr53eYES7UFrXgAAQpLpk1X/+Z//WQkJCUpPT1dxcbHeeOONc567dOlSORwO/+FyhdZ26BuKqrW7vFZx0ZH6u4tzrS4HAIB+L+AgsmTJEtlstm6PzZs3+8//0Y9+pMLCQq1evVqRkZG66667ZBjGWd978eLFcrvd/qOkpKT3v5kFXlh/SJL0t1OGyBE/MIeeAADoSzbjXKngHCorK1VZWdntOfn5+YqNPXPJamlpqVwul9avX6+ZM2ee91oej0cOh0Nut1vJycmBlBl0pScadPnPPpTPkFY/cLlGZiRZXRIAAJYI5P4d8CQGp9Mpp9PZq8I6M8+p80AGit9tKJbPkGYNSyeEAADQQ6bNpty4caM2btyoSy+9VKmpqSoqKtIjjzyiYcOG9ag3JJQ0Nrfp95uKJUn3zMq3thgAAEKIaZNV4+Li9Oqrr+qrX/2qRo0ape9+97saN26c1q5dK7vdbtZlLfHGtiOqaWhRTmqcvjomw+pyAAAIGab1iIwfP14ffPCBWW/fbxiGoec7JqneNTNPkRE2awsCACCE8KyZC/TZwZNLdm+fxpJdAAACQRC5QM9/ckgSS3YBAOgNgsgFqKht0uqd5ZKku2fmW1sMAAAhiCByAd77slw+Q5roStGoTJbsAgAQKILIBXh7+1FJ0g3jMy2uBACA0EQQ6aWK2iZ9drBakvT18VkWVwMAQGgiiPTSn78sl2FIk1wpykmNt7ocAABCEkGkl97+onNYht4QAAB6iyDSCxWeJm081D4scz3zQwAA6DWCSC/8eUf7sMzkXIZlAAC4EASRXniLYRkAAPoEQSRAFZ4mbfIPyxBEAAC4EASRAL375clhmSEpcVaXAwBASCOIBIjVMgAA9B2CSACOeZq06TCbmAEA0FcIIgF4d/tRGYY0JTdF2QzLAABwwQgiAVi797gk6bpx7B0CAEBfIIj0UJvP0OZDJyRJs4Y5La4GAICBgSDSQ7vLPar1tirRHqUxWclWlwMAwIBAEOmhjR1P2p2al6rICJvF1QAAMDAQRHqocxOzS4amWVwJAAADB0GkBwzD8PeIEEQAAOg7BJEeOFhZr8q6ZsVERWhCjsPqcgAAGDAIIj3Q2RsyKSdF9qhIi6sBAGDgIIj0wEbmhwAAYAqCSA909ohcTBABAKBPEUTOo6ymUaUnGhVha1+6CwAA+g5B5Dw6l+1elO1Qoj3K4moAABhYCCLnwbJdAADMQxA5D//8kHyCCAAAfY0g0o0T9c3aV1EnSbo4n/khAAD0NYJINzrnhwwfnKj0RLvF1QAAMPAQRLrB/BAAAMxFEOmG/0F3zA8BAMAUBJFzqPe26ssyjyQ2MgMAwCwEkXPYXV6rNp+hjGS7hqTEWV0OAAADEkHkHA5V1kuSCpyJFlcCAMDARRA5h0NV7UEk35lgcSUAAAxcQQkiXq9XkyZNks1m07Zt24JxyQt2sKNHZKgz3uJKAAAYuIISRB588EFlZ2cH41J9xt8jkk6PCAAAZjE9iLz77rtavXq1li9fbval+oxhGDpU2SBJGsrQDAAApjH1cbLHjh3T3Llz9frrrys+/vxDHF6vV16v1/+1x+Mxs7xzqqxrVp23VTab5EpjaAYAALOY1iNiGIbuuecezZs3T9OmTevRzyxdulQOh8N/uFwus8rrVuewTLYjTrHRkZbUAABAOAg4iCxZskQ2m63bY/PmzXr66afl8Xi0ePHiHr/34sWL5Xa7/UdJSUmg5fWJkxNVGZYBAMBMAQ/NzJ8/X3PmzOn2nPz8fD3xxBPasGGD7PauD4ubNm2a7rjjDr3wwgtn/Jzdbj/jfCt07iGSz4oZAABMFXAQcTqdcjqd5z3vqaee0hNPPOH/uqysTNdee61WrVql6dOnB3rZoOocmhnKZmYAAJjKtMmqubm5Xb5OTGy/qQ8bNkw5OTlmXbZPHPSvmKFHBAAAM7Gz6mkMw9Bh9hABACAoTF2+e6r8/HwZhhGsy/VaRa1XDc1tioywsXQXAACT0SNyms4VMzmpcYqOpHkAADATd9rT+FfMMCwDAIDpCCKnOVjFHiIAAAQLQeQ0J3tEmB8CAIDZCCKn6XzYXT49IgAAmI4gcgqfzzhlMzOCCAAAZiOInKLc0yRvq09RETYNSYmzuhwAAAY8gsgpOueH5KbFK4qluwAAmI677Sk6V8wwPwQAgOAgiJyCPUQAAAgugsgpeNgdAADBRRA5xSGGZgAACCqCSIc2n6Hiqo49RBiaAQAgKAgiHcpqGtXc5lNMZISyWboLAEBQEEQ6dA7L5KbHKzLCZnE1AACEB4JIB1bMAAAQfASRDkWVnVu7s2IGAIBgIYh0KKlun6iaR48IAABBQxDp4GlqlSSlJcRYXAkAAOGDINKhqaVNkhQXHWlxJQAAhA+CSIfG5vYgEksQAQAgaAgiHRo7e0RiCCIAAAQLQaQDQzMAAAQfQaRD59AMQQQAgOAhiEgyDMM/NBMbTZMAABAs3HUlNbf55DPaP49ljggAAEFDEJHU1Ozzf87QDAAAwUMQ0ckVM1ERNkVH0iQAAAQLd12dsnSX3hAAAIKKIKJTNjNjfggAAEFFEBE9IgAAWIUgIjYzAwDAKgQRMTQDAIBVCCI6dWiG5gAAIJi484o5IgAAWIUgolPmiDA0AwBAUBFEdMocEXpEAAAIKlODSH5+vmw2W5dj0aJFZl6yVxiaAQDAGlFmX+Dxxx/X3Llz/V8nJiaafcmAEUQAALCG6UEkKSlJmZmZZl/mgjQ1M0cEAAArmD5H5Kc//anS09M1adIk/eQnP1Fzc7PZlwxYZ48Ic0QAAAguU3tEFixYoClTpig1NVUbN27U4sWLdfDgQf3mN7856/ler1der9f/tcfjMbM8v8YWnySGZgAACLaAe0SWLFlyxgTU04/NmzdLkh544AHNnj1bEyZM0H333adf/OIXWrFihaqqqs763kuXLpXD4fAfLpfrwn67HmpkaAYAAEsE3CMyf/58zZkzp9tz8vPzz/r6jBkzJEn79+9Xenr6Gd9fvHixFi5c6P/a4/EEJYzwrBkAAKwRcBBxOp1yOp29ulhhYaEkKSsr66zft9vtstvtvXrvC8EcEQAArGHaHJFPP/1UGzZs0JVXXimHw6FNmzbpgQce0I033qjc3FyzLtsr7KwKAIA1TAsidrtdq1at0mOPPSav16u8vDzNnTtXDz74oFmX7DX2EQEAwBqmBZEpU6Zow4YNZr19n/LvI0IQAQAgqHjWjE7pEYmhOQAACCbuvGKyKgAAVgn7IOLzGWpiQzMAACwR9kHE2+rzf86qGQAAgivsg0jnsIwkxUYRRAAACCaCSEcQsUdFKCLCZnE1AACEF4IIz5kBAMAyYR9EeM4MAADWCfsgwq6qAABYhyDSzB4iAABYhSDCA+8AALBM2AcR5ogAAGCdsA8iDM0AAGAdgghDMwAAWIYg4h+aCfumAAAg6ML+7tvUzBwRAACsEvZBpLNHJJahGQAAgo4gwqoZAAAsQxBp9kkiiAAAYIWwDyKd+4iwfBcAgOAL+yDC0AwAANYhiDQzWRUAAKsQROgRAQDAMmEfRHjWDAAA1gn7IHJyi/ewbwoAAIIu7O++PPQOAADrEEQYmgEAwDJhH0SaePouAACWCesg0tLmU0ubIYkeEQAArBDWQaSzN0RijggAAFYI6yDSOT/EZpPsUWHdFAAAWCKs775NpzzwzmazWVwNAADhJ6yDCCtmAACwFkFEzA8BAMAq4R1Emlm6CwCAlcI6iPCcGQAArBXWQYQ5IgAAWCu8g0jnc2YYmgEAwBKmB5G3335b06dPV1xcnJxOp2655RazL9ljJ3tEwjqPAQBgmSgz3/yVV17R3Llz9W//9m+66qqrZBiGtm/fbuYlA8IcEQAArGVaEGltbdWCBQu0bNky3Xvvvf7XR40aZdYlA8aqGQAArGXamMTWrVt15MgRRUREaPLkycrKytL111+vHTt2nPNnvF6vPB5Pl8NM7CMCAIC1TAsiRUVFkqQlS5bo4Ycf1ltvvaXU1FTNnj1b1dXVZ/2ZpUuXyuFw+A+Xy2VWeZJYNQMAgNUCDiJLliyRzWbr9ti8ebN8vvbnuDz00EP65je/qalTp2rlypWy2Wz64x//eNb3Xrx4sdxut/8oKSm5sN/uPJgjAgCAtQKeIzJ//nzNmTOn23Py8/NVW1srSRo7dqz/dbvdroKCAhUXF5/15+x2u+x2e6Al9RpzRAAAsFbAQcTpdMrpdJ73vKlTp8put2vPnj269NJLJUktLS06dOiQ8vLyAq/UBMwRAQDAWqatmklOTta8efP06KOPyuVyKS8vT8uWLZMk3XbbbWZdNiCNLe3DRwzNAABgDVP3EVm2bJmioqJ05513qrGxUdOnT9cHH3yg1NRUMy/bY00MzQAAYClTg0h0dLSWL1+u5cuXm3mZXmtqZbIqAABWCuu9zf3PmiGIAABgifAOIi0MzQAAYKWwDiLsIwIAgLXCOoj49xEhiAAAYImwDSKGYZzcRyQmbJsBAABLhe0duLnNJ5/R/jk9IgAAWCNsg0hTs8//OatmAACwRtgGkc5hmehIm6Ijw7YZAACwVNjegXnODAAA1gvfIMKKGQAALBe+QYTNzAAAsFzYBhE2MwMAwHphG0R4zgwAANYL3yDin6watk0AAIDlwvYu3MjQDAAAlgvbINLEZFUAACwXtkGEOSIAAFgvfIMIQzMAAFiOIEIQAQDAMmEbRJqamSMCAIDVwjaI8KwZAACsF8ZBxCeJoRkAAKwUvkGEoRkAACwXtkGEZ80AAGC9sA0izBEBAMB64RtEGJoBAMByYRtEGJoBAMB6YRtE2NAMAADrEURiwrYJAACwXNjehXnoHQAA1gvLIOLzGfK2sqEZAABWC8sg0tTa5v+cVTMAAFgnLINI57CMJMVGEUQAALBKeAaRjomq9qgIRUTYLK4GAIDwFZZBxL+HCMMyAABYKiyDSGMzE1UBAOgPwjOIsJkZAAD9QlgHEfYQAQDAWqYFkY8++kg2m+2sx6ZNm8y6bI+4UuP0g6uG61vTciytAwCAcGczDMMw442bm5tVXV3d5bUf//jH+stf/qKioiLZbOdfreLxeORwOOR2u5WcnGxGmQAAoI8Fcv+OMquImJgYZWZm+r9uaWnRm2++qfnz5/cohAAAgIHPtCByujfffFOVlZW65557znmO1+uV1+v1f+3xeIJQGQAAsErQJquuWLFC1157rVwu1znPWbp0qRwOh//o7lwAABD6Ag4iS5YsOeck1M5j8+bNXX6mtLRU7733nu69995u33vx4sVyu93+o6SkJNDyAABACAl4aGb+/PmaM2dOt+fk5+d3+XrlypVKT0/XjTfe2O3P2e122e32QEsCAAAhKuAg4nQ65XQ6e3y+YRhauXKl7rrrLkVHRwd6OQAAMICZPkfkgw8+0MGDB887LAMAAMKP6UFkxYoVmjVrlsaMGWP2pQAAQIgxffnuSy+9ZPYlAABAiArLZ80AAID+gSACAAAsQxABAACWIYgAAADLBO1ZM73R+WBgnjkDAEDo6Lxvd97Hu9Ovg0htba0k8cwZAABCUG1trRwOR7fn2IyexBWL+Hw+lZWVKSkpSTabrc/e1+PxyOVyqaSkRMnJyX32vjgTbR08tHVw0d7BQ1sHT1+1tWEYqq2tVXZ2tiIiup8F0q97RCIiIpSTk2Pa+ycnJ/NHHSS0dfDQ1sFFewcPbR08fdHW5+sJ6cRkVQAAYBmCCAAAsExYBhG73a5HH31Udrvd6lIGPNo6eGjr4KK9g4e2Dh4r2rpfT1YFAAADW1j2iAAAgP6BIAIAACxDEAEAAJYhiAAAAMsM2CDy7LPPaujQoYqNjdXUqVO1bt26bs9fu3atpk6dqtjYWBUUFOgXv/hFkCoNfYG09auvvqqvfe1rGjRokJKTkzVz5ky99957Qaw2tAX6d93pk08+UVRUlCZNmmRugQNIoG3t9Xr10EMPKS8vT3a7XcOGDdNvf/vbIFUb2gJt6xdffFETJ05UfHy8srKy9J3vfEdVVVVBqjZ0/fWvf9Xf/M3fKDs7WzabTa+//vp5fyYo90ZjAPr9739vREdHG7/+9a+NnTt3GgsWLDASEhKMw4cPn/X8oqIiIz4+3liwYIGxc+dO49e//rURHR1t/OlPfwpy5aEn0LZesGCB8dOf/tTYuHGjsXfvXmPx4sVGdHS0sXXr1iBXHnoCbetONTU1RkFBgXHNNdcYEydODE6xIa43bX3jjTca06dPN9asWWMcPHjQ+Oyzz4xPPvkkiFWHpkDbet26dUZERITxX//1X0ZRUZGxbt0646KLLjJuvvnmIFceet555x3joYceMl555RVDkvHaa691e36w7o0DMohccsklxrx587q8Nnr0aGPRokVnPf/BBx80Ro8e3eW173//+8aMGTNMq3GgCLStz2bs2LHGY4891telDTi9bevbb7/dePjhh41HH32UINJDgbb1u+++azgcDqOqqioY5Q0ogbb1smXLjIKCgi6vPfXUU0ZOTo5pNQ5EPQkiwbo3DrihmebmZm3ZskXXXHNNl9evueYarV+//qw/8+mnn55x/rXXXqvNmzerpaXFtFpDXW/a+nQ+n0+1tbVKS0szo8QBo7dtvXLlSh04cECPPvqo2SUOGL1p6zfffFPTpk3Tz372Mw0ZMkQjR47UD3/4QzU2Ngaj5JDVm7aeNWuWSktL9c4778gwDB07dkx/+tOfdMMNNwSj5LASrHtjv37oXW9UVlaqra1NGRkZXV7PyMhQeXn5WX+mvLz8rOe3traqsrJSWVlZptUbynrT1qf7+c9/rvr6en3rW98yo8QBozdtvW/fPi1atEjr1q1TVNSA+0/dNL1p66KiIn388ceKjY3Va6+9psrKSv3DP/yDqqurmSfSjd609axZs/Tiiy/q9ttvV1NTk1pbW3XjjTfq6aefDkbJYSVY98YB1yPSyWazdfnaMIwzXjvf+Wd7HWcKtK07vfzyy1qyZIlWrVqlwYMHm1XegNLTtm5ra9O3v/1tPfbYYxo5cmSwyhtQAvm79vl8stlsevHFF3XJJZfo61//up588kk9//zz9Ir0QCBtvXPnTv3gBz/QI488oi1btujPf/6zDh48qHnz5gWj1LATjHvjgPvfJKfTqcjIyDPSdEVFxRnJrlNmZuZZz4+KilJ6erpptYa63rR1p1WrVunee+/VH//4R1199dVmljkgBNrWtbW12rx5swoLCzV//nxJ7TdLwzAUFRWl1atX66qrrgpK7aGmN3/XWVlZGjJkSJfHno8ZM0aGYai0tFQjRowwteZQ1Zu2Xrp0qb7yla/oRz/6kSRpwoQJSkhI0GWXXaYnnniCHuw+FKx744DrEYmJidHUqVO1Zs2aLq+vWbNGs2bNOuvPzJw584zzV69erWnTpik6Otq0WkNdb9paau8Jueeee/TSSy8xrttDgbZ1cnKytm/frm3btvmPefPmadSoUdq2bZumT58erNJDTm/+rr/yla+orKxMdXV1/tf27t2riIgI5eTkmFpvKOtNWzc0NCgiouutKzIyUtLJ/1tH3wjavbFPp772E53LwVasWGHs3LnTuP/++42EhATj0KFDhmEYxqJFi4w777zTf37nEqUHHnjA2Llzp7FixQqW7/ZQoG390ksvGVFRUcYzzzxjHD161H/U1NRY9SuEjEDb+nSsmum5QNu6trbWyMnJMW699VZjx44dxtq1a40RI0YY9913n1W/QsgItK1XrlxpREVFGc8++6xx4MAB4+OPPzamTZtmXHLJJVb9CiGjtrbWKCwsNAoLCw1JxpNPPmkUFhb6l0pbdW8ckEHEMAzjmWeeMfLy8oyYmBhjypQpxtq1a/3fu/vuu43Zs2d3Of+jjz4yJk+ebMTExBj5+fnGc889F+SKQ1cgbT179mxD0hnH3XffHfzCQ1Cgf9enIogEJtC23rVrl3H11VcbcXFxRk5OjrFw4UKjoaEhyFWHpkDb+qmnnjLGjh1rxMXFGVlZWcYdd9xhlJaWBrnq0PPhhx92+++vVfdGm2HQlwUAAKwx4OaIAACA0EEQAQAAliGIAAAAyxBEAACAZQgiAADAMgQRAABgGYIIAACwDEEEAABYhiACAAAsQxABAACWIYgAAADLEEQAAIBl/j8JmKyUIQJu3gAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "x = np.arange(0.001,1.0,0.01)\n",
    "y = np.log(x)\n",
    "plt.plot(x,y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "aa2a2274",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.09750000000000003 ## 0.510825457099338\n",
      "0.5975 ## 2.302584092994546\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "# y是softmax函数的输出，对应概率0=0.1,1=0.0 5。。。\n",
    "# t是监督数据，正确解是2\n",
    "y = [0.1,0.05,0.6,0.0,0.05,0.1,0.0,0.1,0.0,0.0]\n",
    "t = [0,0,1,0,0,0,0,0,0,0]\n",
    "\n",
    "def mean_squared_error(y,t):\n",
    "    return 0.5 * np.sum((y - t)**2)\n",
    "\n",
    "def cross_entropy_error(y,t):\n",
    "    delta = 1e-7\n",
    "    return  - np.sum(t * np.log(y + delta))\n",
    "\n",
    "# 2的概率最高，均方误差最小\n",
    "o1 = mean_squared_error(np.array(y),np.array(t))\n",
    "o2 = cross_entropy_error(np.array(y),np.array(t))\n",
    "print(o1,\"##\",o2)\n",
    "\n",
    "y = [0.1,0.05,0.1,0.0,0.05,0.1,0.0,0.6,0.0,0.0]\n",
    "t = [0,0,1,0,0,0,0,0,0,0]\n",
    "\n",
    "# 7的概率最高，与监督数据误差较大\n",
    "o1 = mean_squared_error(np.array(y),np.array(t))\n",
    "o2 = cross_entropy_error(np.array(y),np.array(t))\n",
    "print(o1,\"##\",o2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "16cda8b1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 784)\n",
      "(60000, 10)\n"
     ]
    }
   ],
   "source": [
    "#迷你批次学习 mini-batch\n",
    "import sys,os\n",
    "sys.path.insert(1,'../dataset/')\n",
    "from dataset.mnist import load_mnist\n",
    "(x_train,t_train),(x_test,t_test) = load_mnist(normalize=True,one_hot_label=True)\n",
    "print(x_train.shape) # 训练集大小，训练数据60000个，输入数据是784维的图像数据\n",
    "print(t_train.shape) # 训练集标签大小，监督数据是10维的数据\n",
    "train_size = x_train.shape[0]\n",
    "batch_size = 10\n",
    "batch_mask = np.random.choice(train_size,batch_size) # 随机选择批次数据\n",
    "x_batch = x_train[batch_mask]\n",
    "t_batch = t_train[batch_mask]\n",
    "\n",
    "def cross_entropy_error_one_hot(y,t):\n",
    "    if y.ndim == 1:\n",
    "        t = t.reshape(1,t.size)\n",
    "        y = y.reshape(1,y.size)\n",
    "    batch_size = y.shape[0]\n",
    "    return -np.sum(t * np.log(y + 1e-7))/batch_size\n",
    "\n",
    "# 当监督数据是标签形式(非one-hot表示，而是像“2”“7”这样的标签)\n",
    "# 如[2,7,0,9,4]\n",
    "def cross_entropy_error_non_one_hot(y,t):\n",
    "    if y.ndim == 1:\n",
    "        t = t.reshape(1,t.size)\n",
    "        y = y.reshape(1,y.size)\n",
    "    batch_size = y.shape[0]\n",
    "    return -np.sum(t * np.log(y[np.arange(batch_size,t)] + 1e-7))/batch_size\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c0bf06cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAGwCAYAAACHJU4LAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABBmUlEQVR4nO3deVhVdeLH8c9lFwRcERBE3BdcUNxLbdNsdbTSMlOzGssWc6Z1ZkpnfpNW0zLVZGVpmpZOuWTZpqVoriCouC+ooIIKKhdBLnDv+f1h0liooMK5y/v1PDxPnHvu5XM6XM7Hc7/neyyGYRgCAABwQl5mBwAAADgfigoAAHBaFBUAAOC0KCoAAMBpUVQAAIDToqgAAACnRVEBAABOy8fsAJfD4XDo8OHDCg4OlsViMTsOAACoAMMwlJ+fr8jISHl5XficiUsXlcOHDys6OtrsGAAA4BJkZmYqKirqguu4dFEJDg6WdGZDQ0JCTE4DAAAqwmq1Kjo6uuw4fiEuXVTOftwTEhJCUQEAwMVUZNgGg2kBAIDToqgAAACnRVEBAABOi6ICAACcFkUFAAA4LYoKAABwWhQVAADgtEwvKocOHdK9996runXrKjAwUB07dtSGDRvMjgUAAJyAqRO+nThxQr169dI111yjb7/9VmFhYdq7d69q1aplZiwAAOAkTC0qL7/8sqKjozV9+vSyZY0bNzYvEAAAcCqmfvSzaNEiJSQk6M4771RYWJji4+M1derU865vs9lktVrP+QIAAO7L1KKSnp6uKVOmqHnz5vr+++81ZswYPf7445o5c2a560+aNEmhoaFlX9w5GQAA92YxDMMw64f7+fkpISFBq1evLlv2+OOPKykpSWvWrPnd+jabTTabrez7s3dfzMvL46aEAABcYT9uP6JrWobJy+viNw+sDKvVqtDQ0Aodv009oxIREaE2bdqcs6x169bKyMgod31/f/+yOyVzx2QAAKrOZ+szNHpGsv44a4McDtPOaZhbVHr16qWdO3ees2zXrl2KiYkxKREAAEjef1wvfLlFktQhKvSKn1GpDFOLypNPPqm1a9fqpZde0p49e/Tpp5/qgw8+0NixY82MBQCAx8rKO60xs1JUYjd0U7twjb2mmal5TC0qXbp00YIFC/TZZ58pLi5O//jHP/Tmm29q2LBhZsYCAMAjFZXYNeaTDco5ZVOr8GC9ekcHWSzmnU2RTB5Me7kqMxgHAACcn2EY+tPnmzQ/5ZBqBfrqq0evUnSdwCr5WS4zmBYAADiHaav2a37KIXl7WfSfezpVWUmpLIoKAAAebuXuY/rn4m2SpOdvaq1ezeqZnOhXFBUAADxY+rFTGjs7RQ5DGtwpSvf3amx2pHNQVAAA8FDWohI9MDNZ1qJSdWpUSy8NijN98OxvUVQAAPBAdoehxz5NVfqxAkWEBui94Z3l7+NtdqzfoagAAOCBXv5uhxJ3HVOAr5em3pegsOAAsyOVi6ICAICH+WLDQX2wIl2S9K87OyiuYajJic6PogIAgAdJyTih5+enSZIeu7aZbmkfaXKiC6OoAADgIbLyTuuhmRtUbHeoX5sGevL6FmZHuiiKCgAAHuB0sV0Pzfx1evw3hnQ09WaDFUVRAQDAzRmGoafnbVbaoTzVCfLT1PsSFOTvY3asCqGoAADg5v6zbI++2nRYPl4WvTvMeabHrwiKCgAAbuyHrdn61w+7JEl/vz1O3ZvUNTlR5VBUAABwUzuyrRo3d6Mk6b4eMbqnWyNzA10CigoAAG7oeEGxHpiRrMJiu3o2rau/3dLG7EiXhKICAICbKS516OFZG3TwxGnF1A3Uf+7pJF9v1zzku2ZqAABwXhO/2qp1+46rpr+Ppt6XoNpBfmZHumQUFQAA3MjMNfs1e12GLBbp30M7qkWDYLMjXRaKCgAAbiJx1zFN/GqbJOmp/i11XesGJie6fBQVAADcwO4j+Xp0dorsDkODOjXUw32amh3piqCoAADg4nJP2XT/jCTl20rVpXFtTRrUThaL80+PXxEUFQAAXJit1K4xszYo8/hpNaoTqPeHJ8jfx9vsWFcMRQUAABdlGIaem5+mpP0nFBzgo2kjE1THha/wKQ9FBQAAF/Xu8r2an3JI3l4W/eeeTmoW5tpX+JSHogIAgAv6bkuWXv1+pyRpwm1t1btFfZMTVQ2KCgAALibtYF7ZPXxG9mys4d1jzA1UhSgqAAC4kOy8Ij0wM0lFJQ71aVFff725tdmRqhRFBQAAF1FYXKrRM5J0xGpTiwY19fY98fJx0Xv4VJR7bx0AAG7C4TD05NyN2nrYqrpBfvpoRBeFBPiaHavKUVQAAHABr/6wU99vPSI/by99cF9nRdcJNDtStaCoAADg5D5PztSU5XslSa/c0V6dY+qYnKj6UFQAAHBi69Jz9fyCNEnSY9c208D4hiYnql4UFQAAnNSB3AKNmbVBJXZDN7eL0JPXtzA7UrWjqAAA4ITyCkt0/8dJOlFYog5RofrXnR3k5eUeNxqsDIoKAABOprjUoTGzNmjvsQJFhAZo6n0JquHnPjcarAyKCgAATuTsjQbXpOeqpr+Ppo3sorCQALNjmYaiAgCAE3n7pz2al3LwzI0Gh3VS64gQsyOZiqICAICTWJh6SK8v2SVJ+sftcerjpjcarAyKCgAATmBdeq6e/mKzJOmPvZvonm6NTE7kHCgqAACYbO+xU3rokw0qtjt0U7twPXNjK7MjOQ2KCgAAJso9ZdOo6UnKO12i+Ea19PpdHT3yMuTzoagAAGCSohK7HpyZrIzjhYquU0NT70tQgK9nXoZ8PhQVAABM4HAY+tN/Nykl46RCa/hq+siuqlfT3+xYToeiAgCACV75fqcWp2XJ19ui94d3VrOwmmZHckoUFQAAqtln6zP0XuKvd0Pu3qSuyYmcF0UFAIBqlLjrmP66cIskadz1zfWH+CiTEzk3U4vKhAkTZLFYzvkKDw83MxIAAFVme5ZVY2enyO4wNKhTQz1xXXOzIzk9H7MDtG3bVkuXLi373tub0c4AAPdz+ORpjZqepFO2UnVvUkeTB7WXxcJlyBdjelHx8fHhLAoAwK3lnS7RyOnrlW0tUrOwmnr/3gT5+TD6oiJM/7+0e/duRUZGKjY2VkOHDlV6evp517XZbLJared8AQDgzGyldo35ZIN2HTmlsGB/fTyqi0IDfc2O5TJMLSrdunXTzJkz9f3332vq1KnKzs5Wz549lZubW+76kyZNUmhoaNlXdHR0NScGAKDiHA5Dz3yxWWvScxXk563po7ooqnag2bFcisUwDMPsEGcVFBSoadOmevrppzV+/PjfPW6z2WSz2cq+t1qtio6OVl5enkJCPPs22AAA5/Pydzs0Zfle+XhZNG1kF/XmbsiSzhy/Q0NDK3T8Nn2Myv8KCgpSu3bttHv37nIf9/f3l78/s/YBAJzfJ2sPaMryM3OlTBrUjpJyiUwfo/K/bDabtm/froiICLOjAABwyZZsO6IXvzwzV8r4G1rozgSGKlwqU4vKn//8ZyUmJmrfvn1at26d7rjjDlmtVo0YMcLMWAAAXLLUjBN67LMUOQxpaJdoPXZtM7MjuTRTP/o5ePCg7r77buXk5Kh+/frq3r271q5dq5iYGDNjAQBwSfbnFGj0jGQVlTjUt2V9/d/AOOZKuUymFpU5c+aY+eMBALhick/ZNGL6eh0vKFZcwxD9555O8vF2qhEWLon/gwAAXKbTxXaNnpGsA7mFiqpdQ9NGdlGQv1Ndr+KyKCoAAFwGu8PQY5+lamPmSdUK9NWM+7sqLDjA7Fhug6ICAMAlMgxDExZt1dLtR+Tn46UP70tQ0/o1zY7lVigqAABcond+2qNP1h6QxSK9OaSjEhrXMTuS26GoAABwCeasz9BrS3ZJkibc2lY3tWMOsKpAUQEAoJKWbDui5xekSZLGXtNUI3o2NjeQG6OoAABQCRsOHNejn56Z0O3OzlH6c7+WZkdyaxQVAAAqaPeRfN3/cbJspQ5d2ypMkwa1Y0K3KkZRAQCgArLyTuu+aeuVd7pE8Y1qMaFbNeH/MAAAF5FXWKIR09YrK69ITesHadqILqrh5212LI9AUQEA4AKKSux6YGaSdh05pQYh/ppxf1fVDvIzO5bHoKgAAHAepXaHHvssVUn7Tyg4wEcz7u+qqNqBZsfyKBQVAADKYRiG/vblVi3Z9uuss63CQ8yO5XEoKgAAlOPNpbv12foMeVmkt4Z2VLcmdc2O5JEoKgAA/MastQf07x93S5L+fnucboxj1lmzUFQAAPgf36Rl6YUvt0iSHr+uue7tHmNyIs9GUQEA4Bc/787RuDkb5TCku7tG68nrm5sdyeNRVAAAkLQx86Qe+iRZxXaHBsSF6/8GMuusM6CoAAA83p6j+Ro5fb0Ki+26qlk9vTm0o7y9KCnOgKICAPBoB08U6t4P1+tkYYk6RNfS+8M7y9+HWWedBUUFAOCxck/ZdN9H65VtLVKzsJqaPrKLgvx9zI6F/0FRAQB4pPyiEo2cnqT0nAI1rFVDn4zuqjpMje90KCoAAI9TVGLXQzM3KO1QnuoG+Wnm6K6KCK1hdiyUg6ICAPAopXaHHv8sVWvSc1XT30cfj+qqpvVrmh0L50FRAQB4DMMw9Nz8NP3wy/17pt6XoHZRoWbHwgVQVAAAHmPytzv0+YaD8rJIb98drx5NuX+Ps6OoAAA8wnuJe/X+inRJ0uTB7dW/bbjJiVARFBUAgNv7bH2GJn+7Q5L0/E2tdFdCtMmJUFEUFQCAW1u06bCeX5AmSRrTp6ke6t3U5ESoDIoKAMBtLd12ROPnbpRhSMO6NdIzN7Y0OxIqiaICAHBLq/fk6JFPU1TqMPSH+Ib6x+1x3GTQBVFUAABuJyXjhB6YmaziUoduaNNAr97RXl7cZNAlUVQAAG5le5ZVI6f9eifkt++Ol483hztXxZ4DALiN9GOnNPyjdbIWlapzTG19cF9nBfhyJ2RXRlEBALiFQydP694P1ynnVLHaRIRo2sguCvTjTsiujqICAHB5R/OLNGzqWh3OK1KT+kGaObqrQmv4mh0LVwBFBQDg0k4WFuu+j9Zrf26hGtaqodkPdFO9mv5mx8IVQlEBALisU7ZSjZyepB3Z+QoL9tenD3ZTRGgNs2PhCqKoAABcUlGJXQ/OSNbGzJOqFeirWQ90U0zdILNj4QqjqAAAXE5xqUOPzE7RmvRc1fT30YxRXdWiQbDZsVAFKCoAAJdSanfo8c9S9dOOo/L38dJHIxLUIbqW2bFQRSgqAACXYXcYGv/fTfpua7b8vL009b4EdWtS1+xYqEIUFQCAS3A4DD0zb7MWbTosHy+L3h3WSb1b1Dc7FqoYRQUA4PQMw9DfvtyiLzYclLeXRW/fHa/r2zQwOxaqAUUFAODUDMPQP77ertnrMmSxSK/f1UED2kWYHQvVxGmKyqRJk2SxWDRu3DizowAAnIRhGHrl+52atmqfJOnlQe11e8eGJqdCdXKKopKUlKQPPvhA7du3NzsKAMCJvPXjHk1ZvleS9I+BcbqrS7TJiVDdTC8qp06d0rBhwzR16lTVrl3b7DgAACfxXuJevbF0lyTprze31vDuMSYnghlMLypjx47VzTffrOuvv/6i69psNlmt1nO+AADuZ/qqfZr87Q5J0lP9W+qBq5uYnAhmMfX+13PmzFFKSoqSkpIqtP6kSZM0ceLEKk4FADDTp+syNPGrbZKkx69tprHXNDM5Ecxk2hmVzMxMPfHEE5o1a5YCAgIq9JznnntOeXl5ZV+ZmZlVnBIAUJ3mbTiovyxMkyT9sXcTPXlDC5MTwWwWwzAMM37wwoUL9Yc//EHe3t5ly+x2uywWi7y8vGSz2c55rDxWq1WhoaHKy8tTSEhIVUcGAFShLzce0pNzN8phSCN7NtaLt7aRxWIxOxaqQGWO36Z99HPdddcpLS3tnGWjRo1Sq1at9Mwzz1y0pAAA3MeiTYfLSsrdXaP1wi2UFJxhWlEJDg5WXFzcOcuCgoJUt27d3y0HALivrzcf1rg5qXIY0pCEaP1zYDt5eVFScIbpV/0AADzXN2lZemLOmTMpd3aO0qRBlBScy9Srfn5r+fLlZkcAAFSTb9Oy9NhnqbI7DA3uFKXJg9tTUvA7nFEBAFS777Zkl5WUQfEN9cod7eVNSUE5KCoAgGr1w9ZsPfppikodhm7vGKlX7+xAScF5UVQAANVm6bYjGvtLSbm1Q6Reo6TgIigqAIBq8dOOI3p49gaV2A3d3D5Cb9zVQT7eHIZwYfyGAACq3LKdRzXmk5QzJaVdhP49pCMlBRXCbwkAoEol7jqmP36yQcV2hwbEhevNoZQUVBy/KQCAKrNi1zE9ODNZxaUO9W/bQG/dHS9fSgoqgd8WAECVWLbjqB74paRc37qB3r67EyUFlcZvDADgilu67ciZj3tKHerXpoHeHdZJfj4cclB5TjUzLQDA9X3/yzwpJXZDA+LC+bgHl4WiAgC4Ys5Oi1/qMHRL+wi9MaQjJQWXhaICALgivt58WE/M2Sj7LzPOvnYn86Tg8lFUAACX7cuNh/Tk3DN3QR4U35Bp8XHFUHUBAJdlfsrBspJyZ+coSgquKM6oAAAu2efJmXp63mYZhjS0S7Re+kM7eVFScAVxRgUAcEnmrM8oKynDujWipKBKcEYFAFBps9cd0F8WbJEkjegRowm3tZXFQknBlUdRAQBUysw1+/XCl1slSaN6NdYLt7ShpKDKUFQAABX2fuJeTfp2hyTpwatj9fxNrSkpqFIUFQDARRmGoX//uFtvLt0tSRp7TVP9uV9LSgqqHEUFAHBBhmFo8nc79H5iuiTpqf4tNfaaZiangqegqAAAzsvhMDTxq62aseaAJOlvt7TR6KtiTU4FT0JRAQCUy+4w9Nz8zfpv8kFZLNL/DYzTsG4xZseCh6GoAAB+p8Tu0J/+u0mLNh2Wl0X6150dNKhTlNmx4IEoKgCAc9hK7Xrs01T9sO2IfLws+vfQeN3cPsLsWPBQFBUAQJmiErv++MkGJe46Jj8fL00Z1knXtW5gdix4MIoKAECSVGAr1QMzkrUmPVc1fL019b4EXdW8ntmx4OEoKgAA5Z0u0ajp65WScVI1/X00bWQXdY2tY3YsgKICAJ4u95RNI6av15ZDVoUE+Gjm6G7qGF3L7FiAJIoKAHi0rLzTuvfDddp7rEB1g/z0yehuahMZYnYsoAxFBQA81L6cAt374TodOnlaEaEBmvVANzWtX9PsWMA5KCoA4IG2Z1k1/KP1yjllU2y9IH0yuquiageaHQv4HYoKAHiYDQdOaNT09bIWlap1RIhm3t9V9YP9zY4FlIuiAgAeZOXuY3po5gadLrErIaa2PhrZRaE1fM2OBZwXRQUAPMS3aVl6fE6qSuyGereor/fu7aRAPw4DcG78hgKAB/hvcqaenbdZDkO6uV2E3hjSUX4+XmbHAi6KogIAbu7Dlen6v8XbJUlDEqL10qB28vaymJwKqBiKCgC4KcMw9MbS3Xrrx92SpId6N9FzA1rJYqGkwHVQVADADTkchv7+9TZ9vHq/JOmp/i31SN+mlBS4HIoKALiZ4lKHnv5ikxZuPCxJ+sftbTW8R2NzQwGXiKICAG6ksLhUY2alaMWuY/Lxsuhfd3bQwPiGZscCLhlFBQDcxPGCYo36OEmbMk+qhq+33r23k65pGWZ2LOCyUFQAwA0cPFGo+6atV/qxAtUK9NX0kV0U36i22bGAy1bporJz50599tlnWrlypfbv36/CwkLVr19f8fHx6t+/vwYPHix/f6ZiBoDqsutIvu77aL2yrUWKDA3QzNFd1Sws2OxYwBVhMQzDqMiKqampevrpp7Vy5Ur17NlTXbt2VcOGDVWjRg0dP35cW7Zs0cqVK2W1WvX0009r3LhxVV5YrFarQkNDlZeXp5AQbksOwPMk7z+u+z9OkrWoVM3Damrm6K6KCK1hdizggipz/K7wGZWBAwfqqaee0ty5c1WnTp3zrrdmzRq98cYbeu211/T8889XPDUAoFJ+3H5Ej8xOka3Uoc4xtfXRiATVCvQzOxZwRVX4jEpxcbH8/Cr+BqjI+lOmTNGUKVO0f/9+SVLbtm31wgsvaMCAARX6GZxRAeCpPk/O1LPz02R3GLq2VZj+c08n1fDzNjsWUCGVOX5X+EYPFS0phYWFFV4/KipKkydPVnJyspKTk3Xttdfq9ttv19atWysaCwA8imEYei9xr576YrPsDkODO0Xp/eGdKSlwW5d0R6q+ffvq4MGDv1u+bt06dezYscKvc+utt+qmm25SixYt1KJFC/3zn/9UzZo1tXbt2kuJBQBuzeEw9M/F2zX52x2SpD/2aaJ/3dlevt7cXBDu65J+u0NCQtS+fXvNmTNHkuRwODRhwgT17t1bt9122yUFsdvtmjNnjgoKCtSjR49y17HZbLJared8AYAnKC51aPx/N+rDn/dJkv5yU2s9N6A1U+LD7V3SPCqLFi3Se++9pwceeECLFi3S/v37lZGRocWLF+v666+v1GulpaWpR48eKioqUs2aNbVgwQK1adOm3HUnTZqkiRMnXkpkAHBZ1qISjflkg1bvzZWPl0Wv3NFegzpFmR0LqBYVHkxbnueee04vv/yyfHx8tHz5cvXs2bPSr1FcXKyMjAydPHlS8+bN04cffqjExMRyy4rNZpPNZiv73mq1Kjo6msG0ANxWVt5pjZqepB3Z+Qry89a793ZWnxb1zY4FXJbKDKa9pKJy4sQJPfDAA/rxxx/16quvKjExUQsXLtQrr7yiRx555JKDS9L111+vpk2b6v3337/oulz1A8Cd7czO18jp65WVV6T6wf6aPrKL4hqGmh0LuGxVMo/K/4qLi1NsbKxSU1MVGxurBx98UHPnztUjjzyixYsXa/HixZcUXDozov1/z5oAgCdaszdXD32SrPyiUjWtH6SPR3VVdJ1As2MB1e6SBtOOGTNGK1asUGxsbNmyIUOGaNOmTSouLq7w6zz//PNlU/GnpaXpL3/5i5YvX65hw4ZdSiwAcAuLNh3WiGnrlV9UqoSY2pr3cE9KCjzWZY1RuVyjR4/Wjz/+qKysLIWGhqp9+/Z65plndMMNN1To+Xz0A8CdGIahqSvT9dI3Zy4/HhAXrjeGdFSAL3OkwL1UyUc/GRkZatSoUYVDHDp0SA0bNrzgOh999FGFXw8A3JndYegfX2/Tx6v3S5JG9mysv93SRt5eXH4Mz1bhj366dOmiBx98UOvXrz/vOnl5eZo6dari4uI0f/78KxIQANxdUYldj36aUlZS/nJTa714KyUFkCpxRmX79u166aWXdOONN8rX11cJCQmKjIxUQECATpw4oW3btmnr1q1KSEjQq6++WuH79QCAJztRUKwHZyYr+cAJ+Xl76V93ddBtHSLNjgU4jQqPUdm8ebPatm2rkpISffvtt1qxYoX279+v06dPq169eoqPj1f//v0VFxdX1ZnLMEYFgCvLyC3UyI/XK/1YgYIDfPTB8AT1aFrX7FhAlauSeVS8vb2VnZ2t+vXrq0mTJkpKSlLduua+oSgqAFzVhgMn9NDMZOUWFCsiNEAfj+qqluHBZscCqkWV3D25Vq1aSk9PlyTt379fDofj8lICgIdavDlLd09dq9yCYrWNDNHCsb0oKcB5VHiMyuDBg9WnTx9FRETIYrEoISFB3t7lXzJ3ttAAAH5lGIbeS0zXy9+dufz4+tZh+vfQeAX5X9Lcm4BHqPC744MPPtCgQYO0Z88ePf7443rwwQcVHMy/AACgIkrsDv1t4RbNScqUxOXHQEVVqsbfeOONkqQNGzboiSeeoKgAQAVYi0r0yKwU/bwnR14W6W+3tNGoXrEXfyKAS7vXz/Tp0690DgBwSwdPFGrU9CTtPnpKgX7eevvueF3XuoHZsQCXwQejAFBFNmWe1OgZyco5ZVODEH99NIK7HwOVRVEBgCrw3ZZsjZubqqISh1qFB2v6qC6KCK1hdizA5VBUAOAKMgxDH67cp5e+3S7DkPq2rK937umkmlzZA1wS3jkAcIWU2B164cut+mx9hiTp3u6NNOHWtvLxrvCUVQB+g6ICAFfAiYJiPTx7g9amH5fFcubGgqOvipXFwuXHwOWgqADAZdpz9JRGz0jSgdxCBfl56y2u7AGuGIoKAFyGFbuOaeynKcovKlVU7Rr6aEQXpsMHriCKCgBcAsMwNHPNAf39622yOwwlxNTWe8M7q15Nf7OjAW6FogIAlVRid2jCoq2ave7MoNk7Okfpn3+Ik79P+fc/A3DpKCoAUAknC4v1yOwUrd6bK4tFevbGVnqodxMGzQJVhKICABW099gpPTAjWftyChTk5603h8brhjYMmgWqEkUFACrg5905emT2BlmLStWwVg19OCJBrSNCzI4FuD2KCgBcgGEY+mTtAU386syg2c4xtfU+g2aBakNRAYDzsJXa9cLCrZqbnClJGhTfUC8NaqcAXwbNAtWFogIA5ThqLdKYWRuUknFSXhbpGQbNAqagqADAb2zMPKk/fpKsI1abQgJ89PY9ndSnRX2zYwEeiaICAP9j3oaDem5BmopLHWoWVlNT70tQbL0gs2MBHouiAgCSSu0OvfTNDk1btU+SdH3rBnpjSAcFB/ianAzwbBQVAB7vREGxHv0sRav25EqSHr+uucZd11xeXoxHAcxGUQHg0XZkW/XgzGRlHj+tQD9vvX5XB90YF2F2LAC/oKgA8FjfbcnS+P9uUmGxXdF1amjqfQlqFc4kboAzoagA8DgOh6E3f9ytt37cLUnq1ayu3rm7k2oH+ZmcDMBvUVQAeJS8whKNm5uqZTuPSZJGXxWr5wa0ko+3l8nJAJSHogLAY2w9nKeHZ6Uo43ih/H289NIf2mlw5yizYwG4AIoKAI8wP+WgnpufJlupQ9F1aui9ezurbWSo2bEAXARFBYBbKy516P8Wb9PMNQckSX1b1tebQzqqViDjUQBXQFEB4LaOWIv0yOwUbThwQhLzowCuiKICwC2tS8/V2E9TlXPKpuAAH705pKOua93A7FgAKomiAsCtGIahaav266VvtsvuMNQqPFjv3dtZjblfD+CSKCoA3EZhcamemZemrzYdliTd3jFSkwa1U6Aff+oAV8W7F4BbSD92Sg/PStHOI/ny8bLorze31oiejWWxMB4FcGUUFQAu7+vNh/XMF5tVUGxX/WB/vTusk7o0rmN2LABXAEUFgMuyldr10uLtmvHLpcddY+vonbvjFRYSYHIyAFcKRQWAS8o8XqhHP03RpoN5kqRH+jbV+BtaMBU+4GYoKgBcztJtRzT+vxtlLSpVaA1fvTGkg65txaXHgDuiqABwGaV2h179YafeT0yXJHWMrqV37olXVO1Ak5MBqCqmniOdNGmSunTpouDgYIWFhWngwIHauXOnmZEAOKnsvCLdM3VdWUkZ2bOx/vvHHpQUwM2ZWlQSExM1duxYrV27VkuWLFFpaan69eungoICM2MBcDI/787RzW+t1Pr9x1XT30fvDuukCbe1lZ8P41EAd2cxDMMwO8RZx44dU1hYmBITE9W7d++Lrm+1WhUaGqq8vDyFhIRUQ0IA1cnuMPT2T7v17x93yzCk1hEhendYJ8Uyyyzg0ipz/HaqMSp5eWdG79epU/78BzabTTabrex7q9VaLbkAVL+j+UUaP3eTft6TI0ka2iVaE25rqwBfb5OTAahOTlNUDMPQ+PHjddVVVykuLq7cdSZNmqSJEydWczIA1W3FrmMa/9+NyjlVrABfL/1zYDsN7hxldiwAJnCaj37Gjh2rxYsX6+eff1ZUVPl/kMo7oxIdHc1HP4CbKLE79NoPu/Re4l5JUqvwYL1zT7yahQWbnAzAleRyH/089thjWrRokVasWHHekiJJ/v7+8vf3r8ZkAKpL5vFCPT4nVakZJyVJ93ZvpL/e3IaPegAPZ2pRMQxDjz32mBYsWKDly5crNjbWzDgATPJtWpaenrdZ+UWlCg7w0SuD22tAuwizYwFwAqYWlbFjx+rTTz/Vl19+qeDgYGVnZ0uSQkNDVaNGDTOjAagGRSV2/ePrbZq9LkOSFN+olt4aGq/oOsyNAuAMU8eonO/269OnT9fIkSMv+nwuTwZc156j+Xr001TtyM6XJD38y716fLlXD+D2XGaMipOM4wVQjQzD0OfJB/Xioq06XWJXvZp+ev2ujurdor7Z0QA4IacYTAvAM1iLSvTXBVu0aNNhSdLVzevptbs6KCw4wORkAJwVRQVAtVi/77ienLtRh06elreXRX/q10JjejeVl1f5HwEDgERRAVDFSuwOvfXjbv1n2R45DKlRnUC9ObSjOjWqbXY0AC6AogKgyuzLKdC4uRu1KfOkJOmOzlGacFtb1fTnTw+AiuGvBYAr7uyA2QlfbVVhsV0hAT56aVA73dI+0uxoAFwMRQXAFXWioFjPL0jTt1vOzIvUvUkdvX5XR0XWYm4kAJVHUQFwxazak6Px/92oI1abfLws+nP/lnrw6ibyZsAsgEtEUQFw2Wyldr32wy59sCJdktSkXpD+PTRe7aJCTU4GwNVRVABcll1H8jVuzkZty7JKku7p1kh/vbm1Av348wLg8vGXBMAlsTsMTft5n179YaeKSx2qHeirlwe3V7+24WZHA+BGKCoAKi0jt1B//nyT1u8/Lkm6pmV9vTy4vcJCmGEWwJVFUQFQYYZh6LP1mfq/xdtUWGxXkJ+3/npLGw3tEn3em4wCwOWgqACokKPWIj09b7OW7zwmSerauI7+dWcHNaobaHIyAO6MogLgor7adFh/+3KLThaWyM/HS0/1a6n7r4rlsmMAVY6iAuC8ThQU629fbtHXm7MkSXENQ/T6XR3VokGwyckAeAqKCoByLdt5VM98sVlH823y9rJo7DXN9Ni1zeTr7WV2NAAehKIC4BzWohK9tHi75iRlSpKa1A/SG3d1VIfoWuYGA+CRKCoAyizbeVTPz09TVl6RJGlUr8Z65sZWCvD1NjkZAE9FUQGgvMIS/f3rbZqXclCSFFM3UC8Pbq/uTeqanAyAp6OoAB5u6bYjen5Bmo7m22SxSKN6xuqp/i1Vw4+zKADMR1EBPNSJgmJN/GqrFm48LOnMjQRfuaO9EhrXMTkZAPyKogJ4oO+2ZOmvC7cq55RNXhbpwaub6MkbWjAWBYDToagAHiT3lE0vLNqqxb/Mi9I8rKZeuaO94hvVNjkZAJSPogJ4AMMw9PXmLL24aKuOFxTL28uiMX2a6PHrmsvfh7MoAJwXRQVwc4dOntYLC7foxx1HJUmtwoP16h0d1C4q1ORkAHBxFBXATdkdhmau2a9/fb9TBcV2+Xpb9EjfZhp7TTP5+TC7LADXQFEB3NCObKuenZemjZknJUmdY2pr8qB2as49egC4GIoK4EaKSux668fd+mBFukodhoL9ffT0gFYa1rWRvLjTMQAXRFEB3MTqvTl6fn6a9ucWSpL6t22gibfFKTw0wORkAHDpKCqAiztZWKx/Lt6uzzecmf6+QYi/Jt4Wpxvjwk1OBgCXj6ICuCjDMPTV5iz9/autyjlVLEm6t3sjPX1jK4UE+JqcDgCuDIoK4IL25RTohS+3aOXuHElSs7CamjyoHdPfA3A7FBXAhRSV2DVl+V5NSdyr4lKH/Ly99Mg1TfVw36ZM3AbALVFUABexfOdRvbhoqw78Mlj26ub19Pfb4xRbL8jkZABQdSgqgJPLyjutv3+1Td9uyZZ0ZrDsC7e01U3twmWxcMkxAPdGUQGcVIndoY9X7dcbS3epsNguby+LRvVsrHE3tFBNf966ADwDf+0AJ5S0/7j+umCLdh7Jl3RmZtl/3B6nNpEhJicDgOpFUQGcSO4pmyZ/u6NsTpTagb56bkBr3dE5ipllAXgkigrgBErsDs1ae0CvL9ml/KJSSdLdXaP1dP9Wqh3kZ3I6ADAPRQUw2ao9OZr41VbtOnJKktQmIkT/GBinzjG1TU4GAOajqAAmyTxeqJe+2V52NU/tQF/9uX9LDe3SSN58zAMAkigqQLU7XWzXe4l79V7iXtlKHfKySMO7x+jJG1qoViAf8wDA/6KoANXEMAx9uyVb/1y8XYdOnpYkdW9SRxNua6tW4VzNAwDloagA1WBHtlUTF23TmvRcSVLDWjX0l5tba0Ack7YBwIVQVIAqdLygWP9eukuz1mXI7jDk7+OlMX2aakyfpqrhx715AOBiKCpAFbCV2jVj9X69/dOessuNB8SF6/mbWiu6TqDJ6QDAdXiZ+cNXrFihW2+9VZGRkbJYLFq4cKGZcYDLZhiGvknL0vWvJ+qlb3Yov6hUrSNCNPuBbppyb2dKCgBUkqlnVAoKCtShQweNGjVKgwcPNjMKcNlSM07on4u3K/nACUlSWLC//ty/pQZ3iuJyYwC4RKYWlQEDBmjAgAEVXt9ms8lms5V9b7VaqyIWUCkHTxTqle92atGmw5KkAF8vPdS7qf7Yu4mCuHkgAFwWl/orOmnSJE2cONHsGIAkKb+oRO8u36uPft6n4lKHLBZpUHyUnurfUuGhAWbHAwC34FJF5bnnntP48ePLvrdarYqOjjYxETxRqd2hucmZev2HXcotKJZ0Zj6Uv97cRnENQ01OBwDuxaWKir+/v/z9/c2OAQ9lGIa+25KtV3/YqfRjBZKk2HpBev6m1rq+dRjzoQBAFXCpogKYZfXeHL383U5tyjwp6cx9eR6/rrmGdYuRn4+pF88BgFujqAAXsPVwnl75bqcSdx2TJAX6eeuBq2L1YO8mCg7wNTkdALg/U4vKqVOntGfPnrLv9+3bp40bN6pOnTpq1KiRicng6TJyC/Xakp36cuOZK3l8vCy6p1sjPXZtc9UP5uNHAKguphaV5ORkXXPNNWXfnx0oO2LECH388ccmpYInyzll0zs/7dHsdQdUYjckSbd2iNSfbmihxvWCTE4HAJ7H1KLSt29fGYZhZgRAknTKVqoPV6Zr6op0FRTbJUlXN6+nZ25sxZU8AGAixqjAoxUWl2rmmgN6P3GvThSWSJLaR4Xq2RtbqWezeianAwBQVOCRikrsmr0uQ1OW71HOqTNzoTSpF6Q/9Wupm9qFc6kxADgJigo8iq3UrrlJmfrPsj06Yj1zO4ZGdQL1xHXNdXvHSPl4c6kxADgTigo8Qondoc+TD+qdn3brcF6RJKlhrRp67NpmGtw5Sr4UFABwShQVuLVSu0MLUg/prZ92K/P4aUlSgxB/PXpNM93VJVr+Pt4mJwQAXAhFBW6p1O7Q15uz9NaPu5Wec2a6+3o1/fRw32Ya1q2RAnwpKADgCigqcCsldocWpBzSu8v3aH9uoaQz092P6dNUw3vEKNCPX3kAcCX81YZbKCqx6/MNB/Xe8r06dPLMRzy1A301+qpYjewVq5r+/KoDgCvirzdc2uliuz5dn6EPVuwtu4qnXk1/PdQ7VsO6xSiIggIALo2/4nBJp2yl+mTNAX24Ml25BWfmQYkIDdCYPk01pEs0Y1AAwE1QVOBS8gpL9PHq/Zq2ap/yTp+ZSTa6Tg090reZBnVqyFU8AOBmKCpwCdl5RZq+ap9mr8vQKVupJKlJ/SCN7dtMt3WMZB4UAHBTFBU4td1H8vXBinQt3Hio7G7GLRsE69Frm+mmdhHy9mKqewBwZxQVOB3DMJR84ITeT9yrpduPli3vGltHY/o0Ud8WYfKioACAR6CowGk4HIaWbD+i9xP3KiXjpCTJYpH6twnXQ32aqFOj2uYGBABUO4oKTGcrtWtByiF9sDJd6cfOzCLr5+2lwZ0b6oGrm6hp/ZomJwQAmIWiAtMcLyjWZ+sz9PHq/TqWf2YOlJAAH93bPUYjezVWWHCAyQkBAGajqKDa7czO1/RV+7Qg9ZBspQ5JZ+ZAGX1VrIZ2bcQssgCAMhwRUC0cDkPLdh7VtFX7tGpPbtnydg1DNapXY93SPlJ+PlxiDAA4F0UFVeqUrVRfJGfq49X7y24S6GWRbowL1/29YtU5prYsFq7gAQCUj6KCKpF5vFAzVu/X3KRM5f8yQVtIgI/u7tpIw3vEKKp2oMkJAQCugKKCK8bhMPTznhzNWntAS7cfkePM/GxqUj9Io3o21qBOUdwkEABQKRw1cNlOFBTriw0HNXvdgbKPdyTp6ub1dP9VserTvD4TtAEALglFBZfEMAylZp7UrLUH9PXmLBX/cvVOsL+PBnVqqHu7x6h5g2CTUwIAXB1FBZVSWFyqLzce1qy1B7T1sLVsedvIEN3bPUa3dYjk4x0AwBXDEQUVsvtIvmavy9C8DQfLBsf6+XjplvYRGt49Rh2ja3H1DgDgiqOo4LwKbKVavDlLc5MzteHAibLlMXUDdW+3GN3ROUq1g/xMTAgAcHcUFZzDMAylZJzUf5My9fXmwyootkuSvL0surZVmIZ3j9FVzeoxOBYAUC0oKpAk5ZyyaUHKIc1NztSeo6fKlsfWC9KdCVG6o1OUwkK49w4AoHpRVDyY3WFoxa5jmpuUqaXbj6j0l4lPAny9dFO7CA1JiFbX2DqMPQEAmIai4oF2HcnXgtRDWpBySNnWorLlHaJraUhCtG7tEKHgAF8TEwIAcAZFxUMctRZp0abDmp9ySNuyfr2suHagr/4QH6UhXaLVMpx5TwAAzoWi4sYKbKX6YVu25qcc0qo9OWVT2vt6W9S3ZZgGxTfUta3D5O/jbW5QAADOg6LiZkrtDq3am6uFqYf03ZZsnS6xlz3WOaa2BsY31C3tIrisGADgEigqbsDhODOd/eLNWfpq82Edy7eVPda4bqD+EB+lgfGRiqkbZGJKAAAqj6LiogzD0KaDefp602F9k5alw3m/DoqtHeirWztE6g/xDZkxFgDg0igqLsQwDKUdytPizVn6enOWDp08XfZYTX8f3dCmgW5uF6E+LevL19vLxKQAAFwZFBUnZxiGth62anFalhZvzlLG8cKyxwL9vHV96wa6uX2E+rSorwBfBsUCANwLRcUJ2R2GUjNO6IdtR/TD1mztz/21nNTw9dZ1rcN0S/sI9W0ZRjkBALg1ioqTKCqxa9WeHP2w9Yh+3HFEOaeKyx4L8PXSta3CdHO7SF3Tqr4C/dhtAADPwBHPRCcLi/XTjqP6YesRrdh9TIXFv15KHBzgo+tahalf23D1aVFfQf7sKgCA5+HoV80ycgv1444j+mHrEa3ff1z2s7OwSYoIDVC/Ng3Ur224usbWYUAsAMDjUVSqWFGJXev3Hdfynce0fOdRpecUnPN4q/DgsnLSNjKES4kBAPgfFJUqkHm8UMt3HdPyHUe1em/uObPDentZlBBTWze0aaB+bcLVqG6giUkBAHBuFJUrwFZqV9K+E1q+86iW7TyqvcfOPWsSFuyva1qGqW/L+urVvJ5CuDMxAAAVYnpReffdd/Xqq68qKytLbdu21Ztvvqmrr77a7FgXZHcY2nbYqlV7c7RqT46S9h9XUYmj7HFvL4s6N6qtvq3qq2+LMLWOCOYjHQAALoGpRWXu3LkaN26c3n33XfXq1Uvvv/++BgwYoG3btqlRo0ZmRjuHYRhKzynQ6j05WrUnV2vSc5V3uuScdeoH+6tvi/rq2zJMVzWvp9AanDUBAOByWQzDMC6+WtXo1q2bOnXqpClTppQta926tQYOHKhJkyZd9PlWq1WhoaHKy8tTSEjIFc2WnVekVXtytGpvjlbvyVW2teicx2v6+6h7kzrq2bSeejWrpxYNanLWBACACqjM8du0MyrFxcXasGGDnn322XOW9+vXT6tXry73OTabTTbbr3cGtlqtVZJt+qp9mvjVtnOW+Xl7qXNMbfVqVlc9m9VT+4ah8uHyYQAAqpRpRSUnJ0d2u10NGjQ4Z3mDBg2UnZ1d7nMmTZqkiRMnVnm2uIah8rJI7RqGqmezeurVtJ4SGtdmunoAAKqZ6YNpf/txiWEY5/0I5bnnntP48ePLvrdarYqOjr7imeKjayn1hX6MMwEAwGSmFZV69erJ29v7d2dPjh49+ruzLGf5+/vL39+/yrP5eHsptAYf6wAAYDbTjsZ+fn7q3LmzlixZcs7yJUuWqGfPnialAgAAzsTUj37Gjx+v4cOHKyEhQT169NAHH3ygjIwMjRkzxsxYAADASZhaVIYMGaLc3Fz9/e9/V1ZWluLi4vTNN98oJibGzFgAAMBJmDqPyuWqynlUAABA1ajM8ZsRowAAwGlRVAAAgNOiqAAAAKdFUQEAAE6LogIAAJwWRQUAADgtigoAAHBaFBUAAOC0KCoAAMBpmTqF/uU6O6mu1Wo1OQkAAKios8ftikyO79JFJT8/X5IUHR1tchIAAFBZ+fn5Cg0NveA6Ln2vH4fDocOHDys4OFgWi+WKvrbValV0dLQyMzPd8j5C7r59EtvoDtx9+yS20R24+/ZJV34bDcNQfn6+IiMj5eV14VEoLn1GxcvLS1FRUVX6M0JCQtz2F09y/+2T2EZ34O7bJ7GN7sDdt0+6stt4sTMpZzGYFgAAOC2KCgAAcFoUlfPw9/fXiy++KH9/f7OjVAl33z6JbXQH7r59EtvoDtx9+yRzt9GlB9MCAAD3xhkVAADgtCgqAADAaVFUAACA06KoAAAAp+XRReXdd99VbGysAgIC1LlzZ61cufKC6ycmJqpz584KCAhQkyZN9N5771VT0sqZNGmSunTpouDgYIWFhWngwIHauXPnBZ+zfPlyWSyW333t2LGjmlJXzoQJE36XNTw8/ILPcZX9d1bjxo3L3Sdjx44td31n34crVqzQrbfeqsjISFksFi1cuPCcxw3D0IQJExQZGakaNWqob9++2rp160Vfd968eWrTpo38/f3Vpk0bLViwoIq24OIutI0lJSV65pln1K5dOwUFBSkyMlL33XefDh8+fMHX/Pjjj8vdr0VFRVW8NeW72H4cOXLk77J27979oq/rLPvxYttX3r6wWCx69dVXz/uazrQPK3J8cLb3oscWlblz52rcuHH6y1/+otTUVF199dUaMGCAMjIyyl1/3759uummm3T11VcrNTVVzz//vB5//HHNmzevmpNfXGJiosaOHau1a9dqyZIlKi0tVb9+/VRQUHDR5+7cuVNZWVllX82bN6+GxJembdu252RNS0s777qutP/OSkpKOmf7lixZIkm68847L/g8Z92HBQUF6tChg955551yH3/llVf0+uuv65133lFSUpLCw8N1ww03lN3Tqzxr1qzRkCFDNHz4cG3atEnDhw/XXXfdpXXr1lXVZlzQhbaxsLBQKSkp+tvf/qaUlBTNnz9fu3bt0m233XbR1w0JCTlnn2ZlZSkgIKAqNuGiLrYfJenGG288J+s333xzwdd0pv14se377X6YNm2aLBaLBg8efMHXdZZ9WJHjg9O9Fw0P1bVrV2PMmDHnLGvVqpXx7LPPlrv+008/bbRq1eqcZX/84x+N7t27V1nGK+Xo0aOGJCMxMfG86yxbtsyQZJw4caL6gl2GF1980ejQoUOF13fl/XfWE088YTRt2tRwOBzlPu5K+1CSsWDBgrLvHQ6HER4ebkyePLlsWVFRkREaGmq89957532du+66y7jxxhvPWda/f39j6NChVzxzZf12G8uzfv16Q5Jx4MCB864zffp0IzQ09MqGu0LK28YRI0YYt99+e6Vex1n3Y0X24e23325ce+21F1zHmffhb48Pzvhe9MgzKsXFxdqwYYP69et3zvJ+/fpp9erV5T5nzZo1v1u/f//+Sk5OVklJSZVlvRLy8vIkSXXq1LnouvHx8YqIiNB1112nZcuWVXW0y7J7925FRkYqNjZWQ4cOVXp6+nnXdeX9J535nZ01a5buv//+i96A05X24Vn79u1Tdnb2OfvI399fffr0Oe97Ujr/fr3Qc5xJXl6eLBaLatWqdcH1Tp06pZiYGEVFRemWW25Rampq9QS8RMuXL1dYWJhatGihBx98UEePHr3g+q66H48cOaLFixdr9OjRF13XWffhb48Pzvhe9MiikpOTI7vdrgYNGpyzvEGDBsrOzi73OdnZ2eWuX1paqpycnCrLerkMw9D48eN11VVXKS4u7rzrRURE6IMPPtC8efM0f/58tWzZUtddd51WrFhRjWkrrlu3bpo5c6a+//57TZ06VdnZ2erZs6dyc3PLXd9V999ZCxcu1MmTJzVy5MjzruNq+/B/nX3fVeY9efZ5lX2OsygqKtKzzz6re+6554I3eWvVqpU+/vhjLVq0SJ999pkCAgLUq1cv7d69uxrTVtyAAQM0e/Zs/fTTT3rttdeUlJSka6+9Vjab7bzPcdX9OGPGDAUHB2vQoEEXXM9Z92F5xwdnfC+69N2TL9dv/2VqGMYF/7Va3vrlLXcmjz76qDZv3qyff/75guu1bNlSLVu2LPu+R48eyszM1L/+9S/17t27qmNW2oABA8r+u127durRo4eaNm2qGTNmaPz48eU+xxX331kfffSRBgwYoMjIyPOu42r7sDyVfU9e6nPMVlJSoqFDh8rhcOjdd9+94Lrdu3c/ZzBqr1691KlTJ7399tt66623qjpqpQ0ZMqTsv+Pi4pSQkKCYmBgtXrz4ggd0V9yP06ZN07Bhwy461sRZ9+GFjg/O9F70yDMq9erVk7e39++a3tGjR3/XCM8KDw8vd30fHx/VrVu3yrJejscee0yLFi3SsmXLFBUVVennd+/e3fTGX1FBQUFq167defO64v4768CBA1q6dKkeeOCBSj/XVfbh2Su2KvOePPu8yj7HbCUlJbrrrru0b98+LVmy5IJnU8rj5eWlLl26uMR+lc6c6YuJiblgXlfcjytXrtTOnTsv6X3pDPvwfMcHZ3wvemRR8fPzU+fOncuuojhryZIl6tmzZ7nP6dGjx+/W/+GHH5SQkCBfX98qy3opDMPQo48+qvnz5+unn35SbGzsJb1OamqqIiIirnC6qmGz2bR9+/bz5nWl/fdb06dPV1hYmG6++eZKP9dV9mFsbKzCw8PP2UfFxcVKTEw873tSOv9+vdBzzHS2pOzevVtLly69pJJsGIY2btzoEvtVknJzc5WZmXnBvK62H6UzZzk7d+6sDh06VPq5Zu7Dix0fnPK9eNnDcV3UnDlzDF9fX+Ojjz4ytm3bZowbN84ICgoy9u/fbxiGYTz77LPG8OHDy9ZPT083AgMDjSeffNLYtm2b8dFHHxm+vr7GF198YdYmnNfDDz9shIaGGsuXLzeysrLKvgoLC8vW+e32vfHGG8aCBQuMXbt2GVu2bDGeffZZQ5Ixb948Mzbhov70pz8Zy5cvN9LT0421a9cat9xyixEcHOwW++9/2e12o1GjRsYzzzzzu8dcbR/m5+cbqampRmpqqiHJeP31143U1NSyK14mT55shIaGGvPnzzfS0tKMu+++24iIiDCsVmvZawwfPvycK/NWrVpleHt7G5MnTza2b99uTJ482fDx8THWrl1b7dtnGBfexpKSEuO2224zoqKijI0bN57z3rTZbGWv8dttnDBhgvHdd98Ze/fuNVJTU41Ro0YZPj4+xrp168zYxAtuY35+vvGnP/3JWL16tbFv3z5j2bJlRo8ePYyGDRu6zH682O+pYRhGXl6eERgYaEyZMqXc13DmfViR44OzvRc9tqgYhmH85z//MWJiYgw/Pz+jU6dO51y+O2LECKNPnz7nrL98+XIjPj7e8PPzMxo3bnzeX1KzSSr3a/r06WXr/Hb7Xn75ZaNp06ZGQECAUbt2beOqq64yFi9eXP3hK2jIkCFGRESE4evra0RGRhqDBg0ytm7dWva4K++///X9998bkoydO3f+7jFX24dnL5/+7deIESMMwzhzWeSLL75ohIeHG/7+/kbv3r2NtLS0c16jT58+Zeuf9fnnnxstW7Y0fH19jVatWplazC60jfv27Tvve3PZsmVlr/HbbRw3bpzRqFEjw8/Pz6hfv77Rr18/Y/Xq1dW/cb+40DYWFhYa/fr1M+rXr2/4+voajRo1MkaMGGFkZGSc8xrOvB8v9ntqGIbx/vvvGzVq1DBOnjxZ7ms48z6syPHB2d6Lll+CAwAAOB2PHKMCAABcA0UFAAA4LYoKAABwWhQVAADgtCgqAADAaVFUAACA06KoAAAAp0VRAQAATouiAgAAnBZFBQAAOC2KCgAAcFoUFQBO49ixYwoPD9dLL71UtmzdunXy8/PTDz/8YGIyAGbhpoQAnMo333yjgQMHavXq1WrVqpXi4+N1880368033zQ7GgATUFQAOJ2xY8dq6dKl6tKlizZt2qSkpCQFBASYHQuACSgqAJzO6dOnFRcXp8zMTCUnJ6t9+/ZmRwJgEsaoAHA66enpOnz4sBwOhw4cOGB2HAAm4owKAKdSXFysrl27qmPHjmrVqpVef/11paWlqUGDBmZHA2ACigoAp/LUU0/piy++0KZNm1SzZk1dc801Cg4O1tdff212NAAm4KMfAE5j+fLlevPNN/XJJ58oJCREXl5e+uSTT/Tzzz9rypQpZscDYALOqAAAAKfFGRUAAOC0KCoAAMBpUVQAAIDToqgAAACnRVEBAABOi6ICAACcFkUFAAA4LYoKAABwWhQVAADgtCgqAADAaVFUAACA0/p/HwOOXo8ooG4AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.1999999999990898 0.2999999999986347\n",
      "6.00000000000378\n",
      "7.999999999999119\n",
      "f_x: 2*x\n",
      "f_y: 2*y\n",
      "[6. 8.]\n",
      "[0. 4.]\n",
      "[6. 0.]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([-6.11110793e-10,  8.14814391e-10])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 导数\n",
    "# numericcal_diff:数值微分\n",
    "# 不好的实现(舍入误差，精度问题，真的导数和实现中得到的导数的值在严格意义上并不一致)：\n",
    "def numerical_diff_bad(f,x):\n",
    "    h = 10e-50\n",
    "    return (f(x+h) - f(x))/ h\n",
    "#减少误差，中心差分，计算f在(x+h)和(x-h)之间的差分\n",
    "def numerical_dif(f,x):\n",
    "    h = 1e-4\n",
    "    return (f(x+h) - f(x-h))/(2 * h)\n",
    "\n",
    "# f(x) = 0.01x^2 + 0.1x\n",
    "def fun1(x):\n",
    "    return 0.01*x**2 + 0.1*x\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pylab as plt\n",
    "\n",
    "x = np.arange(0.0,20.0,0.1)\n",
    "y = fun1(x)\n",
    "plt.xlabel('x')\n",
    "plt.ylabel('f(x)')\n",
    "plt.plot(x,y)\n",
    "plt.show()\n",
    "\n",
    "print(numerical_dif(fun1,5),numerical_dif(fun1,10))\n",
    "\n",
    "# f(x0,x1) = x0^2 + x1^2\n",
    "def function_2(x):\n",
    "    return np.sum(x**2)\n",
    "\n",
    "# 1.求x0=3,x1=4时，关于x0的偏导数∂f/∂x0，f=f2_tmp1\n",
    "def f2_tmp1(x0):\n",
    "    return x0*x0 + 4.0**2.0\n",
    "print(numerical_dif(f2_tmp1,3.0))\n",
    "\n",
    "# 2.求x0=3,x1=4时，关于x0的偏导数∂f/∂x1，f=f2_tmp2\n",
    "def f2_tmp2(x1):\n",
    "    return 3.0**2.0 + x1*x1\n",
    "print(numerical_dif(f2_tmp2,4.0))\n",
    "\n",
    "# 导入SymPy库\n",
    "import sympy as sp\n",
    "# 定义符号变量\n",
    "x, y = sp.symbols('x y')\n",
    "# 定义函数\n",
    "f = x ** 2 + y ** 2\n",
    "# 求偏导数\n",
    "f_x = sp.diff(f, x)\n",
    "f_y = sp.diff(f, y)\n",
    "# 输出结果\n",
    "print('f_x:', f_x)\n",
    "print('f_y:', f_y)\n",
    "\n",
    "# 求梯度\n",
    "def numerical_gradient(f,x):\n",
    "    h = 1e-4\n",
    "    grad = np.zeros_like(x) # 生成和x形状相同的数值\n",
    "    for i in range(x.size):\n",
    "        tmp_val = x[i]\n",
    "        \n",
    "        # f(x+h)\n",
    "        x[i] = tmp_val + h\n",
    "        fxh1 = f(x)        # 计算函数值\n",
    "        # f(x-h)\n",
    "        x[i] = tmp_val - h\n",
    "        fxh2 = f(x)        # 计算函数值\n",
    "        \n",
    "        grad[i] = (fxh1 - fxh2)/(2 * h) #求导数\n",
    "        x[i] = tmp_val # 还原值\n",
    "    return grad\n",
    "\n",
    "print(numerical_gradient(function_2,np.array([3.0,4.0])))\n",
    "print(numerical_gradient(function_2,np.array([0.0,2.0])))\n",
    "print(numerical_gradient(function_2,np.array([3.0,0.0])))\n",
    "\n",
    "# 梯度下降法,f为要进行最优化的函数\n",
    "def gradient_descent(f,init_x,lr=0.01,step_num=100):\n",
    "    x = init_x\n",
    "    for i in range(step_num):\n",
    "        grad = numerical_gradient(f,x)\n",
    "        x -= lr * grad\n",
    "    return x\n",
    "\n",
    "init_x = np.array([-3.0,4.0])\n",
    "gradient_descent(function_2,init_x,0.1) #不同的学习率可能产生不同的效果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "78f56e7d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-0.48459566  0.5243312   0.04511345]\n",
      " [-0.96368618 -1.78325219 -0.88296112]]\n",
      "[-1.15807496 -1.29032825 -0.76759694]\n",
      "2\n",
      "[[ 0.17890118  0.15673873 -0.33563992]\n",
      " [ 0.26835177  0.2351081  -0.50345987]]\n",
      "(784, 100)\n",
      "(100,)\n",
      "(100, 10)\n",
      "(10,)\n",
      "(784, 100)\n",
      "(100,)\n",
      "(100, 10)\n",
      "(10,)\n"
     ]
    }
   ],
   "source": [
    "import sys,os\n",
    "sys.path.insert(1,'../common/')\n",
    "sys.path.insert(2,'../dataset/')\n",
    "from common.functions import *\n",
    "from common.gradient import numerical_gradient\n",
    "import numpy as np\n",
    "class simpleNet:\n",
    "    def __init__(self):\n",
    "        self.W = np.random.randn(2,3) # 用高斯分布进行初始化\n",
    "    def predict(self,x):\n",
    "        return np.dot(x,self.W)\n",
    "    def loss(self,x,t):\n",
    "        z = self.predict(x)          # 1.预测值\n",
    "        y = softmax(z)               # 2.进行激活\n",
    "        loss = cross_entropy_error(y,t) # 3.计算预测值与实际值的误差\n",
    "        return loss\n",
    "\n",
    "net = simpleNet()\n",
    "print(net.W)  # 显示权重参数\n",
    "x = np.array([0.6,0.9])\n",
    "p = net.predict(x)\n",
    "print(p)\n",
    "print(np.argmax(p)) # 最大值的索引\n",
    "t = np.array([0,0,1]) # 正确解标签\n",
    "net.loss(x,t)         # 损失值\n",
    "\n",
    "f = lambda w:net.loss(x,t)\n",
    "dW = numerical_gradient(f,net.W)  # 求损失函数对应权重参数W的偏导，当dW中元素变化h时损失函数发生多大变化\n",
    "print(dW)\n",
    "\n",
    "class TwoLayerNet:\n",
    "    # 输入层的神经元数，隐藏层的神经元数，输出层的神经元数，偏置因子\n",
    "    def __init__(self,input_size,hidden_size,output_size,weight_init_std=0.01):\n",
    "        self.params = {}    # 保存神经网络的参数\n",
    "        self.params['W1'] = weight_init_std * np.random.randn(input_size,hidden_size) # 第一层权重\n",
    "        self.params['b1'] = np.zeros(hidden_size)                                     # 第一层偏置\n",
    "        self.params['W2'] = weight_init_std * np.random.randn(hidden_size,output_size)\n",
    "        self.params['b2'] = np.zeros(output_size)\n",
    "        \n",
    "    # 推理，x为图像数据\n",
    "    def predict(self,x):\n",
    "        W1,W2 = self.params['W1'],self.params['W2']\n",
    "        b1,b2 = self.params['b1'],self.params['b2']\n",
    "        \n",
    "        a1 = np.dot(x,W1) + b1\n",
    "        z1 = sigmoid(a1)\n",
    "        \n",
    "        a2 = np.dot(z1,W2) + b2\n",
    "        y = softmax(a2)\n",
    "        return y\n",
    "    \n",
    "    # 计算损失函数的值\n",
    "    def loss(self,x,t): # x:输入数据，t:监督数据\n",
    "        y = self.predict(x)\n",
    "        return cross_entropy_error(y,t)\n",
    "    \n",
    "    # 计算识别精度\n",
    "    def accuracy(self,x,t):\n",
    "        y = self.predict(x)\n",
    "        y = np.argmax(y,axis=1)\n",
    "        t = np.argmax(t,axis=1)\n",
    "        accuracy = np.sum(y==t)/float(x.shape[0])\n",
    "        return accuracy\n",
    "    \n",
    "    # 计算权重参数的精度\n",
    "    def numerical_gradient(self,x,t):\n",
    "        loss_W = lambda W:self.loss(x,t)\n",
    "        grads = {}\n",
    "        grads['W1'] = numerical_gradient(loss_W,self.params['W1']) # 第一层权重的梯度\n",
    "        grads['b1'] = numerical_gradient(loss_W,self.params['b1']) # 第一层偏置的梯度\n",
    "        grads['W2'] = numerical_gradient(loss_W,self.params['W2'])\n",
    "        grads['b2'] = numerical_gradient(loss_W,self.params['b2'])\n",
    "        return grads\n",
    "\n",
    "net = TwoLayerNet(input_size=784, hidden_size=100, output_size=10)\n",
    "print(net.params['W1'].shape) # (784, 100)\n",
    "print(net.params['b1'].shape) # (100,)\n",
    "print(net.params['W2'].shape) # (100, 10)\n",
    "print(net.params['b2'].shape) # (10,)\n",
    "\n",
    "x = np.random.rand(100, 784) # 伪输入数据（100笔）\n",
    "t = np.random.rand(100, 10) # 伪正确解标签（100笔）\n",
    "grads = net.numerical_gradient(x, t) #计算梯度\n",
    "print(grads['W1'].shape) # (784, 100)\n",
    "print(grads['b1'].shape) # (100,)\n",
    "print(grads['W2'].shape) # (100, 10)\n",
    "print(grads['b2'].shape) # (10,)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dce5d6b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 2.298314992863833\n",
      "1 2.289984862948168\n",
      "2 2.2815307094977304\n",
      "3 2.2947122793414088\n",
      "4 2.2960675986984045\n",
      "5 2.2710238135345264\n",
      "6 2.299856690588026\n",
      "7 2.300714214631529\n"
     ]
    }
   ],
   "source": [
    "import sys,os\n",
    "sys.path.insert(1,'../dataset/')\n",
    "from dataset.mnist import load_mnist\n",
    "import numpy as np\n",
    "\n",
    "(x_train,t_train),(x_test,t_test) = load_mnist(normalize=True,one_hot_label=True)\n",
    "train_loss_list = []\n",
    "#超参数\n",
    "iters_num = 100\n",
    "train_size = x_train.shape[0]\n",
    "batch_size = 100\n",
    "learing_rate = 0.1\n",
    "network = TwoLayerNet(input_size=784,hidden_size=50,output_size=10)\n",
    "\n",
    "for i in range(iters_num):\n",
    "    #获取mini-batch\n",
    "    batch_mask = np.random.choice(train_size,batch_size)\n",
    "    x_batch = x_train[batch_mask]\n",
    "    t_batch = t_train[batch_mask]\n",
    "    #计算梯度\n",
    "    grad = network.numerical_gradient(x_batch,t_batch)\n",
    "    #更新参数\n",
    "    for key in ('W1','b1','W2','b2'):\n",
    "        network.params[key] -= learing_rate * grad[key]\n",
    "    loss = network.loss(x_batch,t_batch)\n",
    "    train_loss_list.append(loss)\n",
    "    print(i,loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d23d31c6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
